2024-03-09 07:53:17,734 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-09 07:53:18,067 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-09 07:53:18,188 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-09 07:53:18,253 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-09 07:53:18,260 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-09 07:53:18,277 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=261, ProcessCount=170, AvailableMemoryMB=11721
2024-03-09 07:53:18,280 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-09 07:53:18,280 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-09 07:53:18,408 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1014955897:84 , srv1017018140:84 , srv1021992681:84 , srv1039262012:84 , srv1040578279:84 , srv1069842181:84 , srv1090699692:84 , srv1104875866:84 , srv1113630037:84 , srv1128220323:84 , srv1132331905:84 , srv1136240414:84 , srv1145526335:84 , srv1146730432:84 , srv114872172:84 , srv1150635313:84 , srv1179500201:84 , srv1190200599:84 , srv1222994763:84 , srv1227619995:84 , srv1240384519:84 , srv1241311574:84 , srv124542370:84 , srv1261966242:84 , srv1271864644:84 , srv1274506878:84 , srv1308717306:84 , srv1312021335:84 , srv1327469037:84 , srv135990281:84 , srv140625971:84 , srv1419052394:84 , srv14582472:84 , srv1483075120:84 , srv149957367:84 , srv152665123:84 , srv1545731473:84 , srv1613725772:84 , srv1615059357:84 , srv1657608175:84 , srv1662029691:84 , srv1672238210:84 , srv1678450994:84 , srv1679369651:84 , srv1692433154:84 , srv1703880642:84 , srv1726054442:84 , srv1733256558:84 , srv1765842640:84 , srv1766396415:84 , srv1776817974:84 , srv1804479136:84 , srv1820331122:84 , srv1854009033:84 , srv1859638027:84 , srv1863640495:84 , srv1865166540:84 , srv1873086513:84 , srv1879668483:84 , srv1904295091:84 , srv1913739163:84 , srv1935800653:84 , srv195783889:84 , srv1960819642:84 , srv1964028938:84 , srv1966712830:84 , srv1968118370:84 , srv1979386413:84 , srv1991176040:84 , srv2005002571:84 , srv200861470:84 , srv2030356281:84 , srv2035361364:84 , srv2036432481:84 , srv2037248127:84 , srv2043202101:84 , srv259478234:84 , srv266400143:84 , srv32528928:84 , srv360421058:84 , srv376025506:84 , srv394659930:84 , srv460421470:84 , srv478324739:84 , srv489586620:84 , srv506651218:84 , srv575368120:84 , srv633782799:84 , srv707070900:84 , srv731820449:84 , srv768706628:84 , srv789657395:84 , srv922267294:84 , srv94226053:84 , srv947671228:84 , srv956704228:84 , srv965499646:84 , srv984611985:84 , srv99014999:84 , srv433635737:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-09 07:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-09 07:53:18,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2005002571=69, srv1854009033=53, srv259478234=76, srv731820449=90, srv1483075120=33, srv135990281=29, srv1179500201=16, srv1146730432=13, srv1662029691=40, srv1113630037=8, srv1657608175=39, srv1733256558=47, srv1241311574=21, srv1261966242=23, srv1935800653=61, srv32528928=78, srv956704228=96, srv1672238210=41, srv1312021335=27, srv1859638027=54, srv1904295091=59, srv1968118370=66, srv1145526335=12, srv1879668483=58, srv2037248127=74, srv1017018140=1, srv1678450994=42, srv1014955897=0, srv2043202101=75, srv478324739=84, srv1104875866=7, srv1327469037=28, srv1132331905=10, srv1765842640=48, srv1150635313=15, srv1308717306=26, srv1021992681=2, srv1227619995=19, srv768706628=91, srv124542370=22, srv433635737=82, srv1040578279=4, srv1964028938=64, srv506651218=86, srv1222994763=18, srv149957367=34, srv789657395=92, srv1966712830=65, srv1979386413=67, srv394659930=81, srv1679369651=43, srv2030356281=71, srv360421058=79, srv200861470=70, srv1090699692=6, srv1274506878=25, srv1039262012=3, srv575368120=87, srv1766396415=49, srv1804479136=51, srv1820331122=52, srv633782799=88, srv1692433154=44, srv140625971=30, srv460421470=83, srv1190200599=17, srv2035361364=72, srv376025506=80, srv1419052394=31, srv1726054442=46, srv1865166540=56, srv1776817974=50, srv1069842181=5, srv1240384519=20, srv1991176040=68, srv984611985=98, srv2036432481=73, srv922267294=93, srv195783889=62, srv1703880642=45, srv1271864644=24, srv1615059357=38, srv1136240414=11, srv1960819642=63, srv1913739163=60, srv965499646=97, srv1613725772=37, srv1873086513=57, srv707070900=89, srv94226053=94, srv1863640495=55, srv99014999=99, srv1128220323=9, srv14582472=32, srv152665123=35, srv1545731473=36, srv266400143=77, srv114872172=14, srv489586620=85, srv947671228=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-09 07:53:18,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-09 07:53:18,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-09 07:53:18,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-09 07:53:18,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-09 07:53:18,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-09 07:53:18,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-09 07:53:18,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-09 07:53:18,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-09 07:53:18,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-09 07:53:18,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-09 07:53:18,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-09 07:53:18,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-09 07:53:18,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-09 07:53:18,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-09 07:53:18,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-09 07:53:18,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-09 07:53:18,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-09 07:53:18,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-09 07:53:18,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-09 07:53:18,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-09 07:53:18,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-09 07:53:18,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-09 07:53:18,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-09 07:53:18,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-09 07:53:18,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-09 07:53:18,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-09 07:53:18,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-09 07:53:18,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-09 07:53:18,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-09 07:53:18,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-09 07:53:18,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-09 07:53:18,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-09 07:53:18,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-09 07:53:18,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-09 07:53:18,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-09 07:53:18,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-09 07:53:18,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-09 07:53:18,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-09 07:53:18,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-09 07:53:18,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-09 07:53:18,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-09 07:53:18,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-09 07:53:18,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-09 07:53:18,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-09 07:53:18,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-09 07:53:18,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-09 07:53:18,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-09 07:53:18,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-09 07:53:18,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-09 07:53:18,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-09 07:53:18,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-09 07:53:18,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-09 07:53:18,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-09 07:53:18,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-09 07:53:18,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-09 07:53:18,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-09 07:53:18,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-09 07:53:18,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-09 07:53:18,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-09 07:53:18,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-09 07:53:18,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-09 07:53:18,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-09 07:53:18,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-09 07:53:18,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-09 07:53:18,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-09 07:53:18,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-09 07:53:18,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-09 07:53:18,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-09 07:53:18,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-09 07:53:18,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-09 07:53:18,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-09 07:53:18,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-09 07:53:18,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-09 07:53:18,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-09 07:53:18,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-09 07:53:18,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-09 07:53:18,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-09 07:53:18,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-09 07:53:18,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-09 07:53:18,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-09 07:53:18,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-09 07:53:18,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-09 07:53:18,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-09 07:53:18,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-09 07:53:18,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-09 07:53:18,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-09 07:53:18,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-09 07:53:18,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-09 07:53:18,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-09 07:53:18,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-09 07:53:18,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-09 07:53:18,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-09 07:53:18,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-09 07:53:18,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-09 07:53:18,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-09 07:53:18,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-09 07:53:18,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-09 07:53:18,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-09 07:53:18,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-09 07:53:18,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-09 07:53:18,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-09 07:53:18,968 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:53:18,969 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-09 07:53:18,969 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-09 07:53:19,786 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1229 ms to try 5000 different iterations.  Found a solution that moves 4760 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17147297711793122. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04100283007091908, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05046361936990067, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5288888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04100283007091908, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16666666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23291666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:53:19,803 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1014955897:82 , srv1069842181:82 , srv1483075120:82 , srv1545731473:82 , srv1662029691:82 , srv1672238210:82 , srv1692433154:82 , srv1979386413:82 , srv2005002571:82 , srv266400143:82 , srv922267294:82 , srv1021992681:83 , srv1241311574:83 , srv1327469037:83 , srv1904295091:83 , srv32528928:83 , srv376025506:83 , srv94226053:83 , srv965499646:83 , srv984611985:83 , srv1128220323:84 , srv1146730432:84 , srv1222994763:84 , srv1312021335:84 , srv152665123:84 , srv1865166540:84 , srv1960819642:84 , srv1017018140:85 , srv1040578279:85 , srv1113630037:85 , srv1261966242:85 , srv1308717306:85 , srv1613725772:85 , srv1657608175:85 , srv1703880642:85 , srv1765842640:85 , srv1879668483:85 , srv1913739163:85 , srv1935800653:85 , srv1966712830:85 , srv200861470:85 , srv2036432481:85 , srv789657395:85 , srv956704228:85 , srv99014999:85 , srv1145526335:86 , srv1179500201:86 , srv135990281:86 , srv1678450994:86 , srv1726054442:86 , srv1804479136:86 , srv1991176040:86 , srv2030356281:86 , srv394659930:86 , srv478324739:86 , srv489586620:86 , srv575368120:86 , srv633782799:86 , srv947671228:86 , srv1090699692:87 , srv1240384519:87 , srv1820331122:87 , srv1873086513:87 , srv2037248127:87 , srv460421470:87 , srv506651218:87 , srv707070900:87 , srv731820449:87 , srv1274506878:88 , srv14582472:88 , srv1679369651:88 , srv1733256558:88 , srv1964028938:88 , srv2043202101:88 , srv259478234:88 , srv1039262012:89 , srv1132331905:89 , srv1150635313:89 , srv124542370:89 , srv1271864644:89 , srv1766396415:89 , srv1776817974:89 , srv1854009033:89 , srv1968118370:89 , srv768706628:89 , srv1227619995:90 , srv1615059357:90 , srv1863640495:90 , srv2035361364:90 , srv114872172:91 , srv140625971:91 , srv1419052394:91 , srv1104875866:92 , srv1136240414:92 , srv1859638027:92 , srv195783889:92 , srv360421058:92 , srv149957367:93 , srv1190200599:100 , srv433635737:454 }
2024-03-09 07:53:19,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-09 07:53:19,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2005002571=69, srv1854009033=53, srv259478234=76, srv731820449=90, srv1483075120=33, srv135990281=29, srv1179500201=16, srv1146730432=13, srv1662029691=40, srv1113630037=8, srv1657608175=39, srv1733256558=47, srv1241311574=21, srv1261966242=23, srv1935800653=61, srv32528928=78, srv956704228=96, srv1672238210=41, srv1312021335=27, srv1859638027=54, srv1904295091=59, srv1968118370=66, srv1145526335=12, srv1879668483=58, srv2037248127=74, srv1017018140=1, srv1678450994=42, srv1014955897=0, srv2043202101=75, srv478324739=84, srv1104875866=7, srv1327469037=28, srv1132331905=10, srv1765842640=48, srv1150635313=15, srv1308717306=26, srv1021992681=2, srv1227619995=19, srv768706628=91, srv124542370=22, srv433635737=82, srv1040578279=4, srv1964028938=64, srv506651218=86, srv1222994763=18, srv149957367=34, srv789657395=92, srv1966712830=65, srv1979386413=67, srv394659930=81, srv1679369651=43, srv2030356281=71, srv360421058=79, srv200861470=70, srv1090699692=6, srv1274506878=25, srv1039262012=3, srv575368120=87, srv1766396415=49, srv1804479136=51, srv1820331122=52, srv633782799=88, srv1692433154=44, srv140625971=30, srv460421470=83, srv1190200599=17, srv2035361364=72, srv376025506=80, srv1419052394=31, srv1726054442=46, srv1865166540=56, srv1776817974=50, srv1069842181=5, srv1240384519=20, srv1991176040=68, srv984611985=98, srv2036432481=73, srv922267294=93, srv195783889=62, srv1703880642=45, srv1271864644=24, srv1615059357=38, srv1136240414=11, srv1960819642=63, srv1913739163=60, srv965499646=97, srv1613725772=37, srv1873086513=57, srv707070900=89, srv94226053=94, srv1863640495=55, srv99014999=99, srv1128220323=9, srv14582472=32, srv152665123=35, srv1545731473=36, srv266400143=77, srv114872172=14, srv489586620=85, srv947671228=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-09 07:53:19,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-09 07:53:19,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-09 07:53:19,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-09 07:53:19,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-09 07:53:19,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-09 07:53:19,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-09 07:53:19,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-09 07:53:19,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-09 07:53:19,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-09 07:53:19,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-09 07:53:19,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-09 07:53:19,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-09 07:53:19,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-09 07:53:19,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-09 07:53:19,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-09 07:53:19,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-09 07:53:19,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-09 07:53:19,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-09 07:53:19,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-09 07:53:19,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-09 07:53:19,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-09 07:53:19,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-09 07:53:19,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-09 07:53:19,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-09 07:53:19,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-09 07:53:19,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-09 07:53:19,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-09 07:53:19,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-09 07:53:19,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-09 07:53:19,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-09 07:53:19,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-09 07:53:19,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-09 07:53:19,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-09 07:53:19,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-09 07:53:19,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-09 07:53:19,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-09 07:53:19,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-09 07:53:19,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-09 07:53:19,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-09 07:53:19,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-09 07:53:19,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-09 07:53:19,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-09 07:53:19,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-09 07:53:19,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-09 07:53:19,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-09 07:53:19,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-09 07:53:19,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-09 07:53:19,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-09 07:53:19,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-09 07:53:19,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-09 07:53:19,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-09 07:53:19,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-09 07:53:19,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-09 07:53:19,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-09 07:53:19,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-09 07:53:19,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-09 07:53:19,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-09 07:53:19,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-09 07:53:19,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-09 07:53:19,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-09 07:53:19,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-09 07:53:19,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-09 07:53:19,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-09 07:53:19,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-09 07:53:19,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-09 07:53:19,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-09 07:53:19,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-09 07:53:19,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-09 07:53:19,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-09 07:53:19,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-09 07:53:19,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-09 07:53:19,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-09 07:53:19,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-09 07:53:19,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-09 07:53:19,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-09 07:53:19,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-09 07:53:19,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-09 07:53:19,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-09 07:53:19,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-09 07:53:19,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-09 07:53:19,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-09 07:53:19,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-09 07:53:19,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-09 07:53:19,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-09 07:53:19,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-09 07:53:19,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-09 07:53:19,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-09 07:53:19,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-09 07:53:19,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-09 07:53:19,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-09 07:53:19,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-09 07:53:19,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-09 07:53:19,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-09 07:53:19,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-09 07:53:19,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-09 07:53:19,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-09 07:53:19,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-09 07:53:19,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-09 07:53:19,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-09 07:53:19,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-09 07:53:19,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-09 07:53:20,022 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04100283007091908, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05046361936990067, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04100283007091908, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16666666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23291666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:53:20,023 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-09 07:53:20,023 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17143964688554986, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04100283007091908, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05046361936990067, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04100283007091908, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16666666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23291666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-09 07:53:20,424 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 543 ms to try 5000 different iterations.  Found a solution that moves 2616 regions; Going from a computed imbalance of 0.17143964688554986 to a new imbalance of 0.01303762428965558. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02361462639706167); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02869006953262324, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2906666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02361462639706167); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04025, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:53:20,436 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1190200599:74 , srv195783889:78 , srv1104875866:79 , srv360421058:79 , srv1136240414:80 , srv114872172:80 , srv140625971:81 , srv1419052394:81 , srv149957367:81 , srv1615059357:81 , srv1859638027:81 , srv1968118370:81 , srv1039262012:82 , srv1150635313:82 , srv1227619995:82 , srv259478234:82 , srv1132331905:83 , srv1240384519:83 , srv124542370:83 , srv1271864644:83 , srv1733256558:83 , srv1766396415:83 , srv1820331122:83 , srv1854009033:83 , srv2035361364:83 , srv1679369651:84 , srv1726054442:84 , srv1765842640:84 , srv1776817974:84 , srv1873086513:84 , srv1966712830:84 , srv2037248127:84 , srv2043202101:84 , srv394659930:84 , srv460421470:84 , srv707070900:84 , srv768706628:84 , srv1179500201:85 , srv1274506878:85 , srv135990281:85 , srv14582472:85 , srv1804479136:85 , srv1863640495:85 , srv1964028938:85 , srv575368120:85 , srv731820449:85 , srv947671228:85 , srv1090699692:86 , srv1113630037:86 , srv1261966242:86 , srv1657608175:86 , srv1865166540:86 , srv1879668483:86 , srv1913739163:86 , srv2030356281:86 , srv478324739:86 , srv965499646:86 , srv1145526335:87 , srv1146730432:87 , srv1312021335:87 , srv152665123:87 , srv1613725772:87 , srv1991176040:87 , srv489586620:87 , srv506651218:87 , srv956704228:87 , srv99014999:87 , srv1017018140:88 , srv1021992681:88 , srv1040578279:88 , srv1128220323:88 , srv1222994763:88 , srv1308717306:88 , srv1327469037:88 , srv1935800653:88 , srv1960819642:88 , srv200861470:88 , srv2036432481:88 , srv266400143:88 , srv376025506:88 , srv633782799:88 , srv1241311574:89 , srv1672238210:89 , srv1678450994:89 , srv1692433154:89 , srv2005002571:89 , srv789657395:89 , srv1483075120:90 , srv1545731473:90 , srv1662029691:90 , srv1904295091:90 , srv922267294:90 , srv94226053:90 , srv984611985:90 , srv1014955897:91 , srv1069842181:91 , srv1703880642:91 , srv1979386413:91 , srv32528928:91 , srv433635737:530 }
2024-03-09 07:53:20,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-09 07:53:20,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2005002571=69, srv1854009033=53, srv259478234=76, srv731820449=90, srv1483075120=33, srv135990281=29, srv1179500201=16, srv1146730432=13, srv1662029691=40, srv1113630037=8, srv1657608175=39, srv1733256558=47, srv1241311574=21, srv1261966242=23, srv1935800653=61, srv32528928=78, srv956704228=96, srv1672238210=41, srv1312021335=27, srv1859638027=54, srv1904295091=59, srv1968118370=66, srv1145526335=12, srv1879668483=58, srv2037248127=74, srv1017018140=1, srv1678450994=42, srv1014955897=0, srv2043202101=75, srv478324739=84, srv1104875866=7, srv1327469037=28, srv1132331905=10, srv1765842640=48, srv1150635313=15, srv1308717306=26, srv1021992681=2, srv1227619995=19, srv768706628=91, srv124542370=22, srv433635737=82, srv1040578279=4, srv1964028938=64, srv506651218=86, srv1222994763=18, srv149957367=34, srv789657395=92, srv1966712830=65, srv1979386413=67, srv394659930=81, srv1679369651=43, srv2030356281=71, srv360421058=79, srv200861470=70, srv1090699692=6, srv1274506878=25, srv1039262012=3, srv575368120=87, srv1766396415=49, srv1804479136=51, srv1820331122=52, srv633782799=88, srv1692433154=44, srv140625971=30, srv460421470=83, srv1190200599=17, srv2035361364=72, srv376025506=80, srv1419052394=31, srv1726054442=46, srv1865166540=56, srv1776817974=50, srv1069842181=5, srv1240384519=20, srv1991176040=68, srv984611985=98, srv2036432481=73, srv922267294=93, srv195783889=62, srv1703880642=45, srv1271864644=24, srv1615059357=38, srv1136240414=11, srv1960819642=63, srv1913739163=60, srv965499646=97, srv1613725772=37, srv1873086513=57, srv707070900=89, srv94226053=94, srv1863640495=55, srv99014999=99, srv1128220323=9, srv14582472=32, srv152665123=35, srv1545731473=36, srv266400143=77, srv114872172=14, srv489586620=85, srv947671228=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-09 07:53:20,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-09 07:53:20,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-09 07:53:20,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-09 07:53:20,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-09 07:53:20,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-09 07:53:20,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-09 07:53:20,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-09 07:53:20,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-09 07:53:20,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-09 07:53:20,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-09 07:53:20,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-09 07:53:20,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-09 07:53:20,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-09 07:53:20,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-09 07:53:20,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-09 07:53:20,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-09 07:53:20,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-09 07:53:20,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-09 07:53:20,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-09 07:53:20,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-09 07:53:20,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-09 07:53:20,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-09 07:53:20,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-09 07:53:20,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-09 07:53:20,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-09 07:53:20,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-09 07:53:20,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-09 07:53:20,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-09 07:53:20,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-09 07:53:20,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-09 07:53:20,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-09 07:53:20,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-09 07:53:20,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-09 07:53:20,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-09 07:53:20,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-09 07:53:20,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-09 07:53:20,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-09 07:53:20,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-09 07:53:20,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-09 07:53:20,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-09 07:53:20,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-09 07:53:20,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-09 07:53:20,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-09 07:53:20,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-09 07:53:20,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-09 07:53:20,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-09 07:53:20,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-09 07:53:20,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-09 07:53:20,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-09 07:53:20,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-09 07:53:20,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-09 07:53:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-09 07:53:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-09 07:53:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-09 07:53:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-09 07:53:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-09 07:53:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-09 07:53:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-09 07:53:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-09 07:53:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-09 07:53:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-09 07:53:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-09 07:53:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-09 07:53:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-09 07:53:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-09 07:53:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-09 07:53:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-09 07:53:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-09 07:53:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-09 07:53:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-09 07:53:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-09 07:53:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-09 07:53:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-09 07:53:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-09 07:53:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-09 07:53:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-09 07:53:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-09 07:53:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-09 07:53:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-09 07:53:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-09 07:53:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-09 07:53:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-09 07:53:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-09 07:53:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-09 07:53:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-09 07:53:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-09 07:53:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-09 07:53:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-09 07:53:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-09 07:53:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-09 07:53:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-09 07:53:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-09 07:53:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-09 07:53:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-09 07:53:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-09 07:53:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-09 07:53:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-09 07:53:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-09 07:53:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-09 07:53:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-09 07:53:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-09 07:53:20,616 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02361462639706167); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02869006953262324, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02361462639706167); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04025, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:53:20,616 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-09 07:53:20,616 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013019306666145161, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02361462639706167); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02869006953262324, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02361462639706167); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04025, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-09 07:53:20,889 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 386 ms to try 5000 different iterations.  Found a solution that moves 907 regions; Going from a computed imbalance of 0.013019306666145161 to a new imbalance of 6.002972553682686E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006438288014889025); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005058579404262188); MoveCostFunction : (multiplier=7.0, imbalance=0.10077777777777777, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006438288014889025); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:53:20,890 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1703880642:81 , srv1678450994:82 , srv1132331905:83 , srv1863640495:83 , srv506651218:83 , srv633782799:83 , srv1014955897:84 , srv1017018140:84 , srv1145526335:84 , srv1190200599:84 , srv1241311574:84 , srv149957367:84 , srv1859638027:84 , srv1904295091:84 , srv1964028938:84 , srv2035361364:84 , srv32528928:84 , srv489586620:84 , srv768706628:84 , srv789657395:84 , srv94226053:84 , srv984611985:84 , srv1040578279:85 , srv1069842181:85 , srv1090699692:85 , srv1128220323:85 , srv1136240414:85 , srv124542370:85 , srv1308717306:85 , srv140625971:85 , srv1419052394:85 , srv14582472:85 , srv1483075120:85 , srv1662029691:85 , srv1679369651:85 , srv1776817974:85 , srv1854009033:85 , srv1935800653:85 , srv1960819642:85 , srv1979386413:85 , srv1991176040:85 , srv200861470:85 , srv2030356281:85 , srv2036432481:85 , srv2043202101:85 , srv956704228:85 , srv1021992681:86 , srv1039262012:86 , srv1104875866:86 , srv1146730432:86 , srv114872172:86 , srv1150635313:86 , srv1179500201:86 , srv1227619995:86 , srv1271864644:86 , srv1274506878:86 , srv1312021335:86 , srv1327469037:86 , srv135990281:86 , srv152665123:86 , srv1613725772:86 , srv1615059357:86 , srv1657608175:86 , srv1672238210:86 , srv1692433154:86 , srv1766396415:86 , srv1873086513:86 , srv1913739163:86 , srv2005002571:86 , srv2037248127:86 , srv360421058:86 , srv376025506:86 , srv460421470:86 , srv478324739:86 , srv575368120:86 , srv731820449:86 , srv922267294:86 , srv1113630037:87 , srv1222994763:87 , srv1240384519:87 , srv1261966242:87 , srv1545731473:87 , srv1726054442:87 , srv1733256558:87 , srv1804479136:87 , srv1879668483:87 , srv195783889:87 , srv259478234:87 , srv394659930:87 , srv707070900:87 , srv947671228:87 , srv99014999:87 , srv1820331122:88 , srv1865166540:88 , srv1968118370:88 , srv266400143:88 , srv965499646:88 , srv1966712830:89 , srv1765842640:90 , srv433635737:531 }
2024-03-09 07:53:20,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-09 07:53:20,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2005002571=69, srv1854009033=53, srv259478234=76, srv731820449=90, srv1483075120=33, srv135990281=29, srv1179500201=16, srv1146730432=13, srv1662029691=40, srv1113630037=8, srv1657608175=39, srv1733256558=47, srv1241311574=21, srv1261966242=23, srv1935800653=61, srv32528928=78, srv956704228=96, srv1672238210=41, srv1312021335=27, srv1859638027=54, srv1904295091=59, srv1968118370=66, srv1145526335=12, srv1879668483=58, srv2037248127=74, srv1017018140=1, srv1678450994=42, srv1014955897=0, srv2043202101=75, srv478324739=84, srv1104875866=7, srv1327469037=28, srv1132331905=10, srv1765842640=48, srv1150635313=15, srv1308717306=26, srv1021992681=2, srv1227619995=19, srv768706628=91, srv124542370=22, srv433635737=82, srv1040578279=4, srv1964028938=64, srv506651218=86, srv1222994763=18, srv149957367=34, srv789657395=92, srv1966712830=65, srv1979386413=67, srv394659930=81, srv1679369651=43, srv2030356281=71, srv360421058=79, srv200861470=70, srv1090699692=6, srv1274506878=25, srv1039262012=3, srv575368120=87, srv1766396415=49, srv1804479136=51, srv1820331122=52, srv633782799=88, srv1692433154=44, srv140625971=30, srv460421470=83, srv1190200599=17, srv2035361364=72, srv376025506=80, srv1419052394=31, srv1726054442=46, srv1865166540=56, srv1776817974=50, srv1069842181=5, srv1240384519=20, srv1991176040=68, srv984611985=98, srv2036432481=73, srv922267294=93, srv195783889=62, srv1703880642=45, srv1271864644=24, srv1615059357=38, srv1136240414=11, srv1960819642=63, srv1913739163=60, srv965499646=97, srv1613725772=37, srv1873086513=57, srv707070900=89, srv94226053=94, srv1863640495=55, srv99014999=99, srv1128220323=9, srv14582472=32, srv152665123=35, srv1545731473=36, srv266400143=77, srv114872172=14, srv489586620=85, srv947671228=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-09 07:53:20,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-09 07:53:20,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-09 07:53:20,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-09 07:53:20,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-09 07:53:20,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-09 07:53:20,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-09 07:53:20,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-09 07:53:20,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-09 07:53:20,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-09 07:53:20,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-09 07:53:20,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-09 07:53:20,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-09 07:53:20,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-09 07:53:20,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-09 07:53:20,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-09 07:53:20,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-09 07:53:20,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-09 07:53:20,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-09 07:53:20,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-09 07:53:20,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-09 07:53:20,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-09 07:53:20,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-09 07:53:20,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-09 07:53:20,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-09 07:53:20,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-09 07:53:20,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-09 07:53:20,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-09 07:53:20,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-09 07:53:20,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-09 07:53:20,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-09 07:53:20,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-09 07:53:20,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-09 07:53:20,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-09 07:53:20,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-09 07:53:20,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-09 07:53:20,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-09 07:53:20,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-09 07:53:20,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-09 07:53:20,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-09 07:53:20,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-09 07:53:20,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-09 07:53:20,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-09 07:53:20,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-09 07:53:20,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-09 07:53:20,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-09 07:53:20,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-09 07:53:20,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-09 07:53:20,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-09 07:53:20,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-09 07:53:20,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-09 07:53:20,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-09 07:53:20,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-09 07:53:20,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-09 07:53:20,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-09 07:53:20,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-09 07:53:20,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-09 07:53:20,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-09 07:53:20,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-09 07:53:20,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-09 07:53:20,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-09 07:53:20,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-09 07:53:20,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-09 07:53:20,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-09 07:53:20,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-09 07:53:20,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-09 07:53:20,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-09 07:53:20,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-09 07:53:20,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-09 07:53:20,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-09 07:53:20,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-09 07:53:20,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-09 07:53:20,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-09 07:53:21,017 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006438288014889025); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005058579404262188); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006438288014889025); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:53:21,018 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-09 07:53:21,018 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.939463056266978E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006438288014889025); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005058579404262188); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006438288014889025); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-09 07:53:21,360 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 436 ms to try 5000 different iterations.  Found a solution that moves 257 regions; Going from a computed imbalance of 5.939463056266978E-4 to a new imbalance of 2.676478664543583E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.028555555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:53:21,361 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1222994763:83 , srv1274506878:83 , srv1545731473:83 , srv1765842640:83 , srv1776817974:83 , srv99014999:83 , srv1040578279:84 , srv1069842181:84 , srv1090699692:84 , srv1113630037:84 , srv1227619995:84 , srv1261966242:84 , srv1271864644:84 , srv1308717306:84 , srv14582472:84 , srv149957367:84 , srv1613725772:84 , srv1726054442:84 , srv1766396415:84 , srv1804479136:84 , srv1820331122:84 , srv1863640495:84 , srv1865166540:84 , srv1935800653:84 , srv195783889:84 , srv1966712830:84 , srv1968118370:84 , srv1979386413:84 , srv1991176040:84 , srv200861470:84 , srv266400143:84 , srv32528928:84 , srv478324739:84 , srv707070900:84 , srv731820449:84 , srv789657395:84 , srv922267294:84 , srv94226053:84 , srv947671228:84 , srv1014955897:85 , srv1017018140:85 , srv1021992681:85 , srv1039262012:85 , srv1104875866:85 , srv1128220323:85 , srv1136240414:85 , srv1145526335:85 , srv1146730432:85 , srv114872172:85 , srv1150635313:85 , srv1179500201:85 , srv1190200599:85 , srv1240384519:85 , srv1241311574:85 , srv124542370:85 , srv1312021335:85 , srv1327469037:85 , srv135990281:85 , srv140625971:85 , srv1419052394:85 , srv1483075120:85 , srv152665123:85 , srv1615059357:85 , srv1657608175:85 , srv1662029691:85 , srv1672238210:85 , srv1678450994:85 , srv1679369651:85 , srv1692433154:85 , srv1703880642:85 , srv1733256558:85 , srv1854009033:85 , srv1859638027:85 , srv1873086513:85 , srv1879668483:85 , srv1904295091:85 , srv1913739163:85 , srv1960819642:85 , srv1964028938:85 , srv2005002571:85 , srv2030356281:85 , srv2035361364:85 , srv2036432481:85 , srv2037248127:85 , srv2043202101:85 , srv259478234:85 , srv360421058:85 , srv376025506:85 , srv394659930:85 , srv460421470:85 , srv489586620:85 , srv506651218:85 , srv633782799:85 , srv768706628:85 , srv956704228:85 , srv965499646:85 , srv984611985:85 , srv575368120:86 , srv1132331905:87 , srv433635737:627 }
2024-03-09 07:53:21,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-09 07:53:21,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2005002571=69, srv1854009033=53, srv259478234=76, srv731820449=90, srv1483075120=33, srv135990281=29, srv1179500201=16, srv1146730432=13, srv1662029691=40, srv1113630037=8, srv1657608175=39, srv1733256558=47, srv1241311574=21, srv1261966242=23, srv1935800653=61, srv32528928=78, srv956704228=96, srv1672238210=41, srv1312021335=27, srv1859638027=54, srv1904295091=59, srv1968118370=66, srv1145526335=12, srv1879668483=58, srv2037248127=74, srv1017018140=1, srv1678450994=42, srv1014955897=0, srv2043202101=75, srv478324739=84, srv1104875866=7, srv1327469037=28, srv1132331905=10, srv1765842640=48, srv1150635313=15, srv1308717306=26, srv1021992681=2, srv1227619995=19, srv768706628=91, srv124542370=22, srv433635737=82, srv1040578279=4, srv1964028938=64, srv506651218=86, srv1222994763=18, srv149957367=34, srv789657395=92, srv1966712830=65, srv1979386413=67, srv394659930=81, srv1679369651=43, srv2030356281=71, srv360421058=79, srv200861470=70, srv1090699692=6, srv1274506878=25, srv1039262012=3, srv575368120=87, srv1766396415=49, srv1804479136=51, srv1820331122=52, srv633782799=88, srv1692433154=44, srv140625971=30, srv460421470=83, srv1190200599=17, srv2035361364=72, srv376025506=80, srv1419052394=31, srv1726054442=46, srv1865166540=56, srv1776817974=50, srv1069842181=5, srv1240384519=20, srv1991176040=68, srv984611985=98, srv2036432481=73, srv922267294=93, srv195783889=62, srv1703880642=45, srv1271864644=24, srv1615059357=38, srv1136240414=11, srv1960819642=63, srv1913739163=60, srv965499646=97, srv1613725772=37, srv1873086513=57, srv707070900=89, srv94226053=94, srv1863640495=55, srv99014999=99, srv1128220323=9, srv14582472=32, srv152665123=35, srv1545731473=36, srv266400143=77, srv114872172=14, srv489586620=85, srv947671228=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-09 07:53:21,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-09 07:53:21,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-09 07:53:21,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-09 07:53:21,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-09 07:53:21,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-09 07:53:21,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-09 07:53:21,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-09 07:53:21,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-09 07:53:21,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-09 07:53:21,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-09 07:53:21,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-09 07:53:21,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-09 07:53:21,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-09 07:53:21,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-09 07:53:21,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-09 07:53:21,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-09 07:53:21,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-09 07:53:21,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-09 07:53:21,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-09 07:53:21,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-09 07:53:21,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-09 07:53:21,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-09 07:53:21,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-09 07:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-09 07:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-09 07:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-09 07:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-09 07:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-09 07:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-09 07:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-09 07:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-09 07:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-09 07:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-09 07:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-09 07:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-09 07:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-09 07:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-09 07:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-09 07:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-09 07:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-09 07:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-09 07:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-09 07:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-09 07:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-09 07:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-09 07:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-09 07:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-09 07:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-09 07:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-09 07:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-09 07:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-09 07:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-09 07:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-09 07:53:21,486 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:53:21,486 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-09 07:53:21,487 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.4965234182829804E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-09 07:53:21,722 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 329 ms to try 5000 different iterations.  Found a solution that moves 13 regions; Going from a computed imbalance of 2.4965234182829804E-5 to a new imbalance of 7.780449745930073E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0014444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:53:21,722 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1190200599:83 , srv1733256558:83 , srv1879668483:83 , srv2036432481:83 , srv575368120:83 , srv1014955897:84 , srv1017018140:84 , srv1021992681:84 , srv1039262012:84 , srv1040578279:84 , srv1069842181:84 , srv1090699692:84 , srv1104875866:84 , srv1113630037:84 , srv1128220323:84 , srv1132331905:84 , srv1136240414:84 , srv1145526335:84 , srv1146730432:84 , srv114872172:84 , srv1150635313:84 , srv1179500201:84 , srv1222994763:84 , srv1227619995:84 , srv1240384519:84 , srv124542370:84 , srv1261966242:84 , srv1271864644:84 , srv1274506878:84 , srv1308717306:84 , srv1312021335:84 , srv1327469037:84 , srv135990281:84 , srv140625971:84 , srv1419052394:84 , srv14582472:84 , srv1483075120:84 , srv149957367:84 , srv152665123:84 , srv1545731473:84 , srv1613725772:84 , srv1615059357:84 , srv1657608175:84 , srv1662029691:84 , srv1672238210:84 , srv1678450994:84 , srv1679369651:84 , srv1692433154:84 , srv1703880642:84 , srv1765842640:84 , srv1766396415:84 , srv1804479136:84 , srv1820331122:84 , srv1854009033:84 , srv1859638027:84 , srv1863640495:84 , srv1865166540:84 , srv1873086513:84 , srv1904295091:84 , srv1913739163:84 , srv1935800653:84 , srv1960819642:84 , srv1964028938:84 , srv1966712830:84 , srv1968118370:84 , srv1979386413:84 , srv1991176040:84 , srv2005002571:84 , srv200861470:84 , srv2030356281:84 , srv2035361364:84 , srv2037248127:84 , srv2043202101:84 , srv259478234:84 , srv266400143:84 , srv32528928:84 , srv360421058:84 , srv376025506:84 , srv394659930:84 , srv460421470:84 , srv478324739:84 , srv489586620:84 , srv506651218:84 , srv633782799:84 , srv707070900:84 , srv731820449:84 , srv768706628:84 , srv789657395:84 , srv922267294:84 , srv947671228:84 , srv956704228:84 , srv965499646:84 , srv984611985:84 , srv99014999:84 , srv1241311574:85 , srv1726054442:85 , srv1776817974:85 , srv195783889:85 , srv94226053:85 , srv433635737:684 }
2024-03-09 07:53:21,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-09 07:53:21,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2005002571=69, srv1854009033=53, srv259478234=76, srv731820449=90, srv1483075120=33, srv135990281=29, srv1179500201=16, srv1146730432=13, srv1662029691=40, srv1113630037=8, srv1657608175=39, srv1733256558=47, srv1241311574=21, srv1261966242=23, srv1935800653=61, srv32528928=78, srv956704228=96, srv1672238210=41, srv1312021335=27, srv1859638027=54, srv1904295091=59, srv1968118370=66, srv1145526335=12, srv1879668483=58, srv2037248127=74, srv1017018140=1, srv1678450994=42, srv1014955897=0, srv2043202101=75, srv478324739=84, srv1104875866=7, srv1327469037=28, srv1132331905=10, srv1765842640=48, srv1150635313=15, srv1308717306=26, srv1021992681=2, srv1227619995=19, srv768706628=91, srv124542370=22, srv433635737=82, srv1040578279=4, srv1964028938=64, srv506651218=86, srv1222994763=18, srv149957367=34, srv789657395=92, srv1966712830=65, srv1979386413=67, srv394659930=81, srv1679369651=43, srv2030356281=71, srv360421058=79, srv200861470=70, srv1090699692=6, srv1274506878=25, srv1039262012=3, srv575368120=87, srv1766396415=49, srv1804479136=51, srv1820331122=52, srv633782799=88, srv1692433154=44, srv140625971=30, srv460421470=83, srv1190200599=17, srv2035361364=72, srv376025506=80, srv1419052394=31, srv1726054442=46, srv1865166540=56, srv1776817974=50, srv1069842181=5, srv1240384519=20, srv1991176040=68, srv984611985=98, srv2036432481=73, srv922267294=93, srv195783889=62, srv1703880642=45, srv1271864644=24, srv1615059357=38, srv1136240414=11, srv1960819642=63, srv1913739163=60, srv965499646=97, srv1613725772=37, srv1873086513=57, srv707070900=89, srv94226053=94, srv1863640495=55, srv99014999=99, srv1128220323=9, srv14582472=32, srv152665123=35, srv1545731473=36, srv266400143=77, srv114872172=14, srv489586620=85, srv947671228=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-09 07:53:21,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-09 07:53:21,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-09 07:53:21,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-09 07:53:21,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-09 07:53:21,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-09 07:53:21,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-09 07:53:21,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-09 07:53:21,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-09 07:53:21,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-09 07:53:21,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-09 07:53:21,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-09 07:53:21,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-09 07:53:21,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-09 07:53:21,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-09 07:53:21,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-09 07:53:21,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-09 07:53:21,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-09 07:53:21,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-09 07:53:21,852 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:53:21,853 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1190200599:83 , srv1733256558:83 , srv1879668483:83 , srv2036432481:83 , srv575368120:83 , srv1014955897:84 , srv1017018140:84 , srv1021992681:84 , srv1039262012:84 , srv1040578279:84 , srv1069842181:84 , srv1090699692:84 , srv1104875866:84 , srv1113630037:84 , srv1128220323:84 , srv1132331905:84 , srv1136240414:84 , srv1145526335:84 , srv1146730432:84 , srv114872172:84 , srv1150635313:84 , srv1179500201:84 , srv1222994763:84 , srv1227619995:84 , srv1240384519:84 , srv124542370:84 , srv1261966242:84 , srv1271864644:84 , srv1274506878:84 , srv1308717306:84 , srv1312021335:84 , srv1327469037:84 , srv135990281:84 , srv140625971:84 , srv1419052394:84 , srv14582472:84 , srv1483075120:84 , srv149957367:84 , srv152665123:84 , srv1545731473:84 , srv1613725772:84 , srv1615059357:84 , srv1657608175:84 , srv1662029691:84 , srv1672238210:84 , srv1678450994:84 , srv1679369651:84 , srv1692433154:84 , srv1703880642:84 , srv1765842640:84 , srv1766396415:84 , srv1804479136:84 , srv1820331122:84 , srv1854009033:84 , srv1859638027:84 , srv1863640495:84 , srv1865166540:84 , srv1873086513:84 , srv1904295091:84 , srv1913739163:84 , srv1935800653:84 , srv1960819642:84 , srv1964028938:84 , srv1966712830:84 , srv1968118370:84 , srv1979386413:84 , srv1991176040:84 , srv2005002571:84 , srv200861470:84 , srv2030356281:84 , srv2035361364:84 , srv2037248127:84 , srv2043202101:84 , srv259478234:84 , srv266400143:84 , srv32528928:84 , srv360421058:84 , srv376025506:84 , srv394659930:84 , srv460421470:84 , srv478324739:84 , srv489586620:84 , srv506651218:84 , srv633782799:84 , srv707070900:84 , srv731820449:84 , srv768706628:84 , srv789657395:84 , srv922267294:84 , srv947671228:84 , srv956704228:84 , srv965499646:84 , srv984611985:84 , srv99014999:84 , srv1241311574:85 , srv1726054442:85 , srv1776817974:85 , srv195783889:85 , srv94226053:85 , srv433635737:684 }
2024-03-09 07:53:21,878 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=288 (was 261) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=10976 (was 11721)
2024-03-09 07:53:21,887 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=288, ProcessCount=170, AvailableMemoryMB=10976
2024-03-09 07:53:21,888 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-09 07:53:21,888 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-09 07:53:21,897 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1110356636:84 , srv1220726343:84 , srv1245648856:84 , srv1326699336:84 , srv1339659374:84 , srv1364067435:84 , srv1370692163:84 , srv1544330067:84 , srv1551162955:84 , srv1592337610:84 , srv1594286771:84 , srv1682171535:84 , srv1759737967:84 , srv1853652120:84 , srv1862300397:84 , srv2047791833:84 , srv2066743499:84 , srv227340968:84 , srv261785597:84 , srv346307128:84 , srv388792786:84 , srv401779408:84 , srv509503949:84 , srv625629556:84 , srv658792836:84 , srv682926428:84 , srv725616564:84 , srv811275300:84 , srv99587356:84 , srv2089695625:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-09 07:53:21,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-09 07:53:21,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv261785597=19, srv227340968=18, srv1220726343=1, srv1759737967=12, srv401779408=22, srv1364067435=5, srv1370692163=6, srv625629556=24, srv725616564=27, srv1594286771=10, srv388792786=21, srv1682171535=11, srv1326699336=3, srv1110356636=0, srv1339659374=4, srv2089695625=17, srv1245648856=2, srv346307128=20, srv811275300=28, srv658792836=25, srv682926428=26, srv1544330067=7, srv1853652120=13, srv2047791833=15, srv99587356=29, srv1592337610=9, srv1551162955=8, srv1862300397=14, srv2066743499=16, srv509503949=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-09 07:53:21,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-09 07:53:21,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-09 07:53:21,922 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0975967964831752, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:53:21,922 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991250056758317, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0975967964831752, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-09 07:55:21,910 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5574243 different iterations.  Found a solution that moves 1868 regions; Going from a computed imbalance of 0.991250056758317 to a new imbalance of 4.360005188259462E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6918518518518518, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:55:21,916 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1110356636:90 , srv1220726343:90 , srv1245648856:90 , srv1326699336:90 , srv1339659374:90 , srv1364067435:90 , srv1370692163:90 , srv1544330067:90 , srv1551162955:90 , srv1592337610:90 , srv1594286771:90 , srv1682171535:90 , srv1759737967:90 , srv1853652120:90 , srv1862300397:90 , srv2047791833:90 , srv2066743499:90 , srv2089695625:90 , srv227340968:90 , srv261785597:90 , srv346307128:90 , srv388792786:90 , srv401779408:90 , srv509503949:90 , srv625629556:90 , srv658792836:90 , srv682926428:90 , srv725616564:90 , srv811275300:90 , srv99587356:90 }
2024-03-09 07:55:21,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-09 07:55:21,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv261785597=19, srv227340968=18, srv1220726343=1, srv1759737967=12, srv401779408=22, srv1364067435=5, srv1370692163=6, srv625629556=24, srv725616564=27, srv1594286771=10, srv388792786=21, srv1682171535=11, srv1326699336=3, srv1110356636=0, srv1339659374=4, srv2089695625=17, srv1245648856=2, srv346307128=20, srv811275300=28, srv658792836=25, srv682926428=26, srv1544330067=7, srv1853652120=13, srv2047791833=15, srv99587356=29, srv1592337610=9, srv1551162955=8, srv1862300397=14, srv2066743499=16, srv509503949=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-09 07:55:21,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-09 07:55:21,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-09 07:55:21,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-09 07:55:21,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-09 07:55:21,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-09 07:55:21,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-09 07:55:21,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-09 07:55:21,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-09 07:55:21,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-09 07:55:21,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-09 07:55:21,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-09 07:55:21,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-09 07:55:21,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-09 07:55:21,933 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-09 07:55:21,933 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1110356636:90 , srv1220726343:90 , srv1245648856:90 , srv1326699336:90 , srv1339659374:90 , srv1364067435:90 , srv1370692163:90 , srv1544330067:90 , srv1551162955:90 , srv1592337610:90 , srv1594286771:90 , srv1682171535:90 , srv1759737967:90 , srv1853652120:90 , srv1862300397:90 , srv2047791833:90 , srv2066743499:90 , srv2089695625:90 , srv227340968:90 , srv261785597:90 , srv346307128:90 , srv388792786:90 , srv401779408:90 , srv509503949:90 , srv625629556:90 , srv658792836:90 , srv682926428:90 , srv725616564:90 , srv811275300:90 , srv99587356:90 }
2024-03-09 07:55:21,948 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=239 (was 288), ProcessCount=168 (was 170), AvailableMemoryMB=10211 (was 10976)