2024-04-27 05:00:48,538 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-04-27 05:00:48,777 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-04-27 05:00:48,867 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-04-27 05:00:48,920 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-04-27 05:00:48,926 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-04-27 05:00:48,939 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=805, ProcessCount=180, AvailableMemoryMB=2404
2024-04-27 05:00:48,941 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-04-27 05:00:48,942 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-04-27 05:00:49,059 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv101190152:84 , srv1126969502:84 , srv1134051180:84 , srv1139040454:84 , srv1207502077:84 , srv1242674123:84 , srv1260783733:84 , srv1356708065:84 , srv1378959108:84 , srv138012263:84 , srv1385484922:84 , srv1400570999:84 , srv1424921584:84 , srv1455821722:84 , srv1466790716:84 , srv1478853152:84 , srv1482755053:84 , srv1493006602:84 , srv151603779:84 , srv1524574489:84 , srv1543053003:84 , srv1545997570:84 , srv156303569:84 , srv1571643727:84 , srv1576983427:84 , srv1635109997:84 , srv1658949280:84 , srv1690458557:84 , srv1699973579:84 , srv1706164386:84 , srv1731658509:84 , srv1735173038:84 , srv173965216:84 , srv1748513716:84 , srv1751681885:84 , srv1775134971:84 , srv1784247114:84 , srv1785325479:84 , srv178755440:84 , srv1790825918:84 , srv1792697605:84 , srv1793672086:84 , srv179712365:84 , srv1836847074:84 , srv1847900061:84 , srv1855427942:84 , srv1861654460:84 , srv1893978183:84 , srv1953908190:84 , srv1967344145:84 , srv2025105754:84 , srv2031331345:84 , srv2031582397:84 , srv2059973197:84 , srv2073132768:84 , srv2099602712:84 , srv2099738448:84 , srv2103981846:84 , srv2143230089:84 , srv229508970:84 , srv231356411:84 , srv240232607:84 , srv262498661:84 , srv310579981:84 , srv316083350:84 , srv340945826:84 , srv350370741:84 , srv351510048:84 , srv435903674:84 , srv450016811:84 , srv450607070:84 , srv453080142:84 , srv472940307:84 , srv517399557:84 , srv519098315:84 , srv578905085:84 , srv583939679:84 , srv59814179:84 , srv661107405:84 , srv6707197:84 , srv694106689:84 , srv714574819:84 , srv719845614:84 , srv731342035:84 , srv731737728:84 , srv744308897:84 , srv748759115:84 , srv753180772:84 , srv786539235:84 , srv807795879:84 , srv810616215:84 , srv823487622:84 , srv851007990:84 , srv930897651:84 , srv936731526:84 , srv940594851:84 , srv942559318:84 , srv964075399:84 , srv968011415:84 , srv1293423040:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-04-27 05:00:49,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-27 05:00:49,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1576983427=25, srv661107405=79, srv519098315=75, srv59814179=78, srv1893978183=48, srv1466790716=15, srv340945826=66, srv1855427942=46, srv1543053003=21, srv1790825918=40, srv964075399=98, srv968011415=99, srv786539235=89, srv940594851=96, srv1571643727=24, srv1260783733=6, srv173965216=33, srv1751681885=35, srv1126969502=1, srv1658949280=27, srv178755440=39, srv1482755053=17, srv583939679=77, srv1493006602=18, srv1775134971=36, srv2099738448=57, srv310579981=64, srv2025105754=51, srv101190152=0, srv262498661=63, srv240232607=62, srv1139040454=3, srv1967344145=50, srv936731526=95, srv316083350=65, srv942559318=97, srv1847900061=45, srv472940307=73, srv1293423040=7, srv694106689=81, srv1784247114=37, srv2073132768=55, srv231356411=61, srv1861654460=47, srv1207502077=4, srv930897651=94, srv1385484922=11, srv453080142=72, srv1690458557=28, srv1731658509=31, srv1748513716=34, srv753180772=88, srv1735173038=32, srv1134051180=2, srv2031582397=53, srv1356708065=8, srv350370741=67, srv748759115=87, srv517399557=74, srv2103981846=58, srv151603779=19, srv1793672086=42, srv229508970=60, srv2099602712=56, srv1836847074=44, srv1524574489=20, srv1378959108=9, srv1400570999=12, srv807795879=90, srv450607070=71, srv810616215=91, srv450016811=70, srv1792697605=41, srv138012263=10, srv2031331345=52, srv823487622=92, srv578905085=76, srv1953908190=49, srv1785325479=38, srv2143230089=59, srv351510048=68, srv1478853152=16, srv1635109997=26, srv1706164386=30, srv6707197=80, srv1455821722=14, srv719845614=83, srv1242674123=5, srv2059973197=54, srv744308897=86, srv731737728=85, srv1424921584=13, srv731342035=84, srv714574819=82, srv179712365=43, srv1545997570=22, srv156303569=23, srv1699973579=29, srv435903674=69, srv851007990=93} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-27 05:00:49,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-27 05:00:49,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-27 05:00:49,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-27 05:00:49,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-27 05:00:49,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-27 05:00:49,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-27 05:00:49,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-27 05:00:49,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-27 05:00:49,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-27 05:00:49,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-27 05:00:49,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-27 05:00:49,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-27 05:00:49,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-27 05:00:49,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-27 05:00:49,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-27 05:00:49,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-27 05:00:49,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-27 05:00:49,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-27 05:00:49,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-27 05:00:49,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-27 05:00:49,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-27 05:00:49,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-27 05:00:49,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-27 05:00:49,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-27 05:00:49,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-27 05:00:49,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-27 05:00:49,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-27 05:00:49,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-27 05:00:49,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-27 05:00:49,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-27 05:00:49,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-27 05:00:49,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-27 05:00:49,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-27 05:00:49,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-27 05:00:49,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-27 05:00:49,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-27 05:00:49,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-27 05:00:49,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-27 05:00:49,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-27 05:00:49,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-27 05:00:49,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-27 05:00:49,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-27 05:00:49,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-27 05:00:49,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-27 05:00:49,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-27 05:00:49,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-27 05:00:49,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-27 05:00:49,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-27 05:00:49,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-27 05:00:49,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-27 05:00:49,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-27 05:00:49,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-27 05:00:49,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-27 05:00:49,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-27 05:00:49,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-27 05:00:49,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-27 05:00:49,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-27 05:00:49,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-27 05:00:49,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-27 05:00:49,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-27 05:00:49,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-27 05:00:49,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-27 05:00:49,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-27 05:00:49,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-27 05:00:49,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-27 05:00:49,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-27 05:00:49,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-27 05:00:49,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-27 05:00:49,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-27 05:00:49,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-27 05:00:49,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-27 05:00:49,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-27 05:00:49,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-27 05:00:49,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-27 05:00:49,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-27 05:00:49,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-27 05:00:49,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-27 05:00:49,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-27 05:00:49,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-27 05:00:49,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-27 05:00:49,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-27 05:00:49,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-27 05:00:49,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-27 05:00:49,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-27 05:00:49,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-27 05:00:49,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-27 05:00:49,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-27 05:00:49,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-27 05:00:49,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-27 05:00:49,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-27 05:00:49,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-27 05:00:49,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-27 05:00:49,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-27 05:00:49,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-27 05:00:49,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-27 05:00:49,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-27 05:00:49,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-27 05:00:49,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-27 05:00:49,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-27 05:00:49,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-27 05:00:49,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-27 05:00:49,275 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-27 05:00:49,275 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-27 05:00:49,276 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-27 05:00:49,516 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 354 ms to try 5000 different iterations.  Found a solution that moves 4681 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18063757172277445. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041762516017449715, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05572499414034637, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5201111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041762516017449715, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.176, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-27 05:00:49,536 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1126969502:82 , srv1543053003:82 , srv173965216:82 , srv1775134971:82 , srv2031582397:82 , srv229508970:82 , srv310579981:82 , srv351510048:82 , srv753180772:82 , srv1139040454:83 , srv1242674123:83 , srv1784247114:83 , srv1836847074:83 , srv316083350:83 , srv930897651:83 , srv101190152:84 , srv1385484922:84 , srv1400570999:84 , srv1576983427:84 , srv1847900061:84 , srv1967344145:84 , srv2103981846:84 , srv435903674:84 , srv450607070:84 , srv744308897:84 , srv1134051180:85 , srv1260783733:85 , srv138012263:85 , srv1424921584:85 , srv1482755053:85 , srv1524574489:85 , srv156303569:85 , srv1690458557:85 , srv1699973579:85 , srv1735173038:85 , srv1748513716:85 , srv1785325479:85 , srv1790825918:85 , srv2059973197:85 , srv2143230089:85 , srv262498661:85 , srv583939679:85 , srv661107405:85 , srv964075399:85 , srv968011415:85 , srv1466790716:86 , srv1493006602:86 , srv1658949280:86 , srv1751681885:86 , srv1855427942:86 , srv1953908190:86 , srv2031331345:86 , srv2099602712:86 , srv340945826:86 , srv472940307:86 , srv519098315:86 , srv59814179:86 , srv6707197:86 , srv786539235:86 , srv823487622:86 , srv936731526:86 , srv1356708065:87 , srv1455821722:87 , srv1478853152:87 , srv151603779:87 , srv1635109997:87 , srv1861654460:87 , srv2025105754:87 , srv450016811:87 , srv453080142:87 , srv714574819:87 , srv731737728:87 , srv807795879:87 , srv810616215:87 , srv1571643727:88 , srv178755440:88 , srv1893978183:88 , srv2073132768:88 , srv231356411:88 , srv350370741:88 , srv578905085:88 , srv942559318:88 , srv1731658509:89 , srv179712365:89 , srv1207502077:90 , srv1378959108:90 , srv1706164386:90 , srv1793672086:90 , srv240232607:90 , srv1792697605:91 , srv2099738448:91 , srv748759115:91 , srv940594851:91 , srv694106689:92 , srv731342035:92 , srv851007990:92 , srv517399557:94 , srv1545997570:95 , srv719845614:95 , srv1293423040:461 }
2024-04-27 05:00:49,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-27 05:00:49,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1576983427=25, srv661107405=79, srv519098315=75, srv59814179=78, srv1893978183=48, srv1466790716=15, srv340945826=66, srv1855427942=46, srv1543053003=21, srv1790825918=40, srv964075399=98, srv968011415=99, srv786539235=89, srv940594851=96, srv1571643727=24, srv1260783733=6, srv173965216=33, srv1751681885=35, srv1126969502=1, srv1658949280=27, srv178755440=39, srv1482755053=17, srv583939679=77, srv1493006602=18, srv1775134971=36, srv2099738448=57, srv310579981=64, srv2025105754=51, srv101190152=0, srv262498661=63, srv240232607=62, srv1139040454=3, srv1967344145=50, srv936731526=95, srv316083350=65, srv942559318=97, srv1847900061=45, srv472940307=73, srv1293423040=7, srv694106689=81, srv1784247114=37, srv2073132768=55, srv231356411=61, srv1861654460=47, srv1207502077=4, srv930897651=94, srv1385484922=11, srv453080142=72, srv1690458557=28, srv1731658509=31, srv1748513716=34, srv753180772=88, srv1735173038=32, srv1134051180=2, srv2031582397=53, srv1356708065=8, srv350370741=67, srv748759115=87, srv517399557=74, srv2103981846=58, srv151603779=19, srv1793672086=42, srv229508970=60, srv2099602712=56, srv1836847074=44, srv1524574489=20, srv1378959108=9, srv1400570999=12, srv807795879=90, srv450607070=71, srv810616215=91, srv450016811=70, srv1792697605=41, srv138012263=10, srv2031331345=52, srv823487622=92, srv578905085=76, srv1953908190=49, srv1785325479=38, srv2143230089=59, srv351510048=68, srv1478853152=16, srv1635109997=26, srv1706164386=30, srv6707197=80, srv1455821722=14, srv719845614=83, srv1242674123=5, srv2059973197=54, srv744308897=86, srv731737728=85, srv1424921584=13, srv731342035=84, srv714574819=82, srv179712365=43, srv1545997570=22, srv156303569=23, srv1699973579=29, srv435903674=69, srv851007990=93} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-27 05:00:49,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-27 05:00:49,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-27 05:00:49,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-27 05:00:49,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-27 05:00:49,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-27 05:00:49,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-27 05:00:49,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-27 05:00:49,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-27 05:00:49,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-27 05:00:49,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-27 05:00:49,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-27 05:00:49,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-27 05:00:49,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-27 05:00:49,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-27 05:00:49,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-27 05:00:49,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-27 05:00:49,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-27 05:00:49,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-27 05:00:49,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-27 05:00:49,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-27 05:00:49,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-27 05:00:49,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-27 05:00:49,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-27 05:00:49,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-27 05:00:49,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-27 05:00:49,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-27 05:00:49,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-27 05:00:49,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-27 05:00:49,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-27 05:00:49,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-27 05:00:49,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-27 05:00:49,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-27 05:00:49,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-27 05:00:49,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-27 05:00:49,664 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041762516017449715, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05572499414034637, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041762516017449715, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.176, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-27 05:00:49,664 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-27 05:00:49,664 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18060479466021626, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041762516017449715, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05572499414034637, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041762516017449715, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.176, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-27 05:00:49,826 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 243 ms to try 5000 different iterations.  Found a solution that moves 2609 regions; Going from a computed imbalance of 0.18060479466021626 to a new imbalance of 0.014752899186368105. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0212831902286175); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03391744119268852, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28988888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0212831902286175); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-27 05:00:49,834 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1545997570:79 , srv1792697605:79 , srv517399557:79 , srv694106689:80 , srv1378959108:81 , srv2099738448:81 , srv719845614:81 , srv1207502077:82 , srv179712365:82 , srv748759115:82 , srv851007990:82 , srv940594851:82 , srv1706164386:83 , srv1793672086:83 , srv1893978183:83 , srv578905085:83 , srv1134051180:84 , srv1356708065:84 , srv1478853152:84 , srv151603779:84 , srv1524574489:84 , srv178755440:84 , srv2031331345:84 , srv2073132768:84 , srv350370741:84 , srv519098315:84 , srv731737728:84 , srv807795879:84 , srv936731526:84 , srv942559318:84 , srv1466790716:85 , srv1571643727:85 , srv1731658509:85 , srv231356411:85 , srv240232607:85 , srv340945826:85 , srv450016811:85 , srv453080142:85 , srv786539235:85 , srv823487622:85 , srv1493006602:86 , srv156303569:86 , srv1635109997:86 , srv1699973579:86 , srv1735173038:86 , srv1748513716:86 , srv1790825918:86 , srv1953908190:86 , srv2025105754:86 , srv2059973197:86 , srv435903674:86 , srv583939679:86 , srv59814179:86 , srv714574819:86 , srv744308897:86 , srv968011415:86 , srv1260783733:87 , srv1385484922:87 , srv1400570999:87 , srv1455821722:87 , srv1658949280:87 , srv1751681885:87 , srv1847900061:87 , srv1855427942:87 , srv1861654460:87 , srv472940307:87 , srv661107405:87 , srv731342035:87 , srv810616215:87 , srv964075399:87 , srv101190152:88 , srv1242674123:88 , srv138012263:88 , srv1424921584:88 , srv1543053003:88 , srv1576983427:88 , srv1784247114:88 , srv1785325479:88 , srv1836847074:88 , srv2099602712:88 , srv2103981846:88 , srv2143230089:88 , srv262498661:88 , srv450607070:88 , srv930897651:88 , srv1126969502:89 , srv1482755053:89 , srv173965216:89 , srv229508970:89 , srv310579981:89 , srv6707197:89 , srv753180772:89 , srv1690458557:90 , srv1775134971:90 , srv2031582397:90 , srv316083350:90 , srv351510048:90 , srv1139040454:91 , srv1967344145:92 , srv1293423040:502 }
2024-04-27 05:00:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-27 05:00:49,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1576983427=25, srv661107405=79, srv519098315=75, srv59814179=78, srv1893978183=48, srv1466790716=15, srv340945826=66, srv1855427942=46, srv1543053003=21, srv1790825918=40, srv964075399=98, srv968011415=99, srv786539235=89, srv940594851=96, srv1571643727=24, srv1260783733=6, srv173965216=33, srv1751681885=35, srv1126969502=1, srv1658949280=27, srv178755440=39, srv1482755053=17, srv583939679=77, srv1493006602=18, srv1775134971=36, srv2099738448=57, srv310579981=64, srv2025105754=51, srv101190152=0, srv262498661=63, srv240232607=62, srv1139040454=3, srv1967344145=50, srv936731526=95, srv316083350=65, srv942559318=97, srv1847900061=45, srv472940307=73, srv1293423040=7, srv694106689=81, srv1784247114=37, srv2073132768=55, srv231356411=61, srv1861654460=47, srv1207502077=4, srv930897651=94, srv1385484922=11, srv453080142=72, srv1690458557=28, srv1731658509=31, srv1748513716=34, srv753180772=88, srv1735173038=32, srv1134051180=2, srv2031582397=53, srv1356708065=8, srv350370741=67, srv748759115=87, srv517399557=74, srv2103981846=58, srv151603779=19, srv1793672086=42, srv229508970=60, srv2099602712=56, srv1836847074=44, srv1524574489=20, srv1378959108=9, srv1400570999=12, srv807795879=90, srv450607070=71, srv810616215=91, srv450016811=70, srv1792697605=41, srv138012263=10, srv2031331345=52, srv823487622=92, srv578905085=76, srv1953908190=49, srv1785325479=38, srv2143230089=59, srv351510048=68, srv1478853152=16, srv1635109997=26, srv1706164386=30, srv6707197=80, srv1455821722=14, srv719845614=83, srv1242674123=5, srv2059973197=54, srv744308897=86, srv731737728=85, srv1424921584=13, srv731342035=84, srv714574819=82, srv179712365=43, srv1545997570=22, srv156303569=23, srv1699973579=29, srv435903674=69, srv851007990=93} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-27 05:00:49,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-27 05:00:49,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-27 05:00:49,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-27 05:00:49,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-27 05:00:49,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-27 05:00:49,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-27 05:00:49,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-27 05:00:49,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-27 05:00:49,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-27 05:00:49,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-27 05:00:49,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-27 05:00:49,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-27 05:00:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-27 05:00:49,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-27 05:00:49,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-27 05:00:49,944 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0212831902286175); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03391744119268852, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0212831902286175); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-27 05:00:49,944 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-27 05:00:49,944 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014734630577905307, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0212831902286175); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03391744119268852, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0212831902286175); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-27 05:00:50,094 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 222 ms to try 5000 different iterations.  Found a solution that moves 939 regions; Going from a computed imbalance of 0.014734630577905307 to a new imbalance of 5.032196146539296E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006205525135896712); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010358338378419612); MoveCostFunction : (multiplier=7.0, imbalance=0.10433333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006205525135896712); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-27 05:00:50,095 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv731342035:78 , srv719845614:81 , srv1690458557:82 , srv1967344145:82 , srv240232607:82 , srv6707197:82 , srv1139040454:83 , srv1455821722:83 , srv1482755053:83 , srv1545997570:83 , srv1635109997:84 , srv1658949280:84 , srv1706164386:84 , srv1731658509:84 , srv1751681885:84 , srv1785325479:84 , srv1793672086:84 , srv1861654460:84 , srv2025105754:84 , srv2099602712:84 , srv2143230089:84 , srv231356411:84 , srv316083350:84 , srv472940307:84 , srv714574819:84 , srv851007990:84 , srv101190152:85 , srv1260783733:85 , srv138012263:85 , srv1424921584:85 , srv1493006602:85 , srv1571643727:85 , srv1775134971:85 , srv178755440:85 , srv1855427942:85 , srv2073132768:85 , srv2099738448:85 , srv2103981846:85 , srv262498661:85 , srv351510048:85 , srv450607070:85 , srv453080142:85 , srv517399557:85 , srv59814179:85 , srv661107405:85 , srv694106689:85 , srv810616215:85 , srv940594851:85 , srv964075399:85 , srv1385484922:86 , srv1478853152:86 , srv151603779:86 , srv1576983427:86 , srv1748513716:86 , srv1784247114:86 , srv179712365:86 , srv1836847074:86 , srv1893978183:86 , srv1953908190:86 , srv2031582397:86 , srv229508970:86 , srv310579981:86 , srv350370741:86 , srv578905085:86 , srv583939679:86 , srv731737728:86 , srv748759115:86 , srv753180772:86 , srv823487622:86 , srv942559318:86 , srv968011415:86 , srv1126969502:87 , srv1207502077:87 , srv1242674123:87 , srv1356708065:87 , srv1378959108:87 , srv1400570999:87 , srv1466790716:87 , srv156303569:87 , srv1699973579:87 , srv1735173038:87 , srv173965216:87 , srv1790825918:87 , srv1792697605:87 , srv1847900061:87 , srv2031331345:87 , srv2059973197:87 , srv340945826:87 , srv435903674:87 , srv450016811:87 , srv519098315:87 , srv744308897:87 , srv786539235:87 , srv807795879:87 , srv930897651:87 , srv1134051180:88 , srv1524574489:88 , srv1543053003:88 , srv936731526:88 , srv1293423040:550 }
2024-04-27 05:00:50,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-27 05:00:50,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1576983427=25, srv661107405=79, srv519098315=75, srv59814179=78, srv1893978183=48, srv1466790716=15, srv340945826=66, srv1855427942=46, srv1543053003=21, srv1790825918=40, srv964075399=98, srv968011415=99, srv786539235=89, srv940594851=96, srv1571643727=24, srv1260783733=6, srv173965216=33, srv1751681885=35, srv1126969502=1, srv1658949280=27, srv178755440=39, srv1482755053=17, srv583939679=77, srv1493006602=18, srv1775134971=36, srv2099738448=57, srv310579981=64, srv2025105754=51, srv101190152=0, srv262498661=63, srv240232607=62, srv1139040454=3, srv1967344145=50, srv936731526=95, srv316083350=65, srv942559318=97, srv1847900061=45, srv472940307=73, srv1293423040=7, srv694106689=81, srv1784247114=37, srv2073132768=55, srv231356411=61, srv1861654460=47, srv1207502077=4, srv930897651=94, srv1385484922=11, srv453080142=72, srv1690458557=28, srv1731658509=31, srv1748513716=34, srv753180772=88, srv1735173038=32, srv1134051180=2, srv2031582397=53, srv1356708065=8, srv350370741=67, srv748759115=87, srv517399557=74, srv2103981846=58, srv151603779=19, srv1793672086=42, srv229508970=60, srv2099602712=56, srv1836847074=44, srv1524574489=20, srv1378959108=9, srv1400570999=12, srv807795879=90, srv450607070=71, srv810616215=91, srv450016811=70, srv1792697605=41, srv138012263=10, srv2031331345=52, srv823487622=92, srv578905085=76, srv1953908190=49, srv1785325479=38, srv2143230089=59, srv351510048=68, srv1478853152=16, srv1635109997=26, srv1706164386=30, srv6707197=80, srv1455821722=14, srv719845614=83, srv1242674123=5, srv2059973197=54, srv744308897=86, srv731737728=85, srv1424921584=13, srv731342035=84, srv714574819=82, srv179712365=43, srv1545997570=22, srv156303569=23, srv1699973579=29, srv435903674=69, srv851007990=93} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-27 05:00:50,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-27 05:00:50,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-27 05:00:50,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-27 05:00:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-27 05:00:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-27 05:00:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-27 05:00:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-27 05:00:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-27 05:00:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-27 05:00:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-27 05:00:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-27 05:00:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-27 05:00:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-27 05:00:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-27 05:00:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-27 05:00:50,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-27 05:00:50,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-27 05:00:50,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-27 05:00:50,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-27 05:00:50,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-27 05:00:50,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-27 05:00:50,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-27 05:00:50,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-27 05:00:50,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-27 05:00:50,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-27 05:00:50,191 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006205525135896712); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010358338378419612); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006205525135896712); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-27 05:00:50,192 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-27 05:00:50,192 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.966445961232407E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006205525135896712); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010358338378419612); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006205525135896712); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-27 05:00:50,410 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 288 ms to try 5000 different iterations.  Found a solution that moves 276 regions; Going from a computed imbalance of 4.966445961232407E-4 to a new imbalance of 1.8523517796211373E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.030666666666666665, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-27 05:00:50,411 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1207502077:83 , srv1466790716:83 , srv450016811:83 , srv748759115:83 , srv810616215:83 , srv1126969502:84 , srv1356708065:84 , srv1378959108:84 , srv138012263:84 , srv1400570999:84 , srv1424921584:84 , srv1543053003:84 , srv156303569:84 , srv1576983427:84 , srv1699973579:84 , srv1731658509:84 , srv1735173038:84 , srv173965216:84 , srv1784247114:84 , srv1790825918:84 , srv1847900061:84 , srv1855427942:84 , srv1861654460:84 , srv1953908190:84 , srv1967344145:84 , srv2031582397:84 , srv2059973197:84 , srv2099602712:84 , srv262498661:84 , srv340945826:84 , srv350370741:84 , srv450607070:84 , srv517399557:84 , srv519098315:84 , srv753180772:84 , srv786539235:84 , srv807795879:84 , srv851007990:84 , srv930897651:84 , srv936731526:84 , srv940594851:84 , srv942559318:84 , srv101190152:85 , srv1134051180:85 , srv1139040454:85 , srv1242674123:85 , srv1260783733:85 , srv1385484922:85 , srv1455821722:85 , srv1478853152:85 , srv1482755053:85 , srv151603779:85 , srv1524574489:85 , srv1545997570:85 , srv1571643727:85 , srv1658949280:85 , srv1690458557:85 , srv1706164386:85 , srv1748513716:85 , srv1751681885:85 , srv1775134971:85 , srv1785325479:85 , srv178755440:85 , srv1792697605:85 , srv1793672086:85 , srv179712365:85 , srv1836847074:85 , srv1893978183:85 , srv2025105754:85 , srv2073132768:85 , srv2099738448:85 , srv2103981846:85 , srv2143230089:85 , srv229508970:85 , srv231356411:85 , srv240232607:85 , srv310579981:85 , srv316083350:85 , srv351510048:85 , srv435903674:85 , srv453080142:85 , srv472940307:85 , srv578905085:85 , srv583939679:85 , srv59814179:85 , srv661107405:85 , srv6707197:85 , srv694106689:85 , srv714574819:85 , srv719845614:85 , srv731342035:85 , srv731737728:85 , srv744308897:85 , srv823487622:85 , srv964075399:85 , srv968011415:85 , srv1493006602:86 , srv1635109997:86 , srv2031331345:86 , srv1293423040:629 }
2024-04-27 05:00:50,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-27 05:00:50,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1576983427=25, srv661107405=79, srv519098315=75, srv59814179=78, srv1893978183=48, srv1466790716=15, srv340945826=66, srv1855427942=46, srv1543053003=21, srv1790825918=40, srv964075399=98, srv968011415=99, srv786539235=89, srv940594851=96, srv1571643727=24, srv1260783733=6, srv173965216=33, srv1751681885=35, srv1126969502=1, srv1658949280=27, srv178755440=39, srv1482755053=17, srv583939679=77, srv1493006602=18, srv1775134971=36, srv2099738448=57, srv310579981=64, srv2025105754=51, srv101190152=0, srv262498661=63, srv240232607=62, srv1139040454=3, srv1967344145=50, srv936731526=95, srv316083350=65, srv942559318=97, srv1847900061=45, srv472940307=73, srv1293423040=7, srv694106689=81, srv1784247114=37, srv2073132768=55, srv231356411=61, srv1861654460=47, srv1207502077=4, srv930897651=94, srv1385484922=11, srv453080142=72, srv1690458557=28, srv1731658509=31, srv1748513716=34, srv753180772=88, srv1735173038=32, srv1134051180=2, srv2031582397=53, srv1356708065=8, srv350370741=67, srv748759115=87, srv517399557=74, srv2103981846=58, srv151603779=19, srv1793672086=42, srv229508970=60, srv2099602712=56, srv1836847074=44, srv1524574489=20, srv1378959108=9, srv1400570999=12, srv807795879=90, srv450607070=71, srv810616215=91, srv450016811=70, srv1792697605=41, srv138012263=10, srv2031331345=52, srv823487622=92, srv578905085=76, srv1953908190=49, srv1785325479=38, srv2143230089=59, srv351510048=68, srv1478853152=16, srv1635109997=26, srv1706164386=30, srv6707197=80, srv1455821722=14, srv719845614=83, srv1242674123=5, srv2059973197=54, srv744308897=86, srv731737728=85, srv1424921584=13, srv731342035=84, srv714574819=82, srv179712365=43, srv1545997570=22, srv156303569=23, srv1699973579=29, srv435903674=69, srv851007990=93} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-27 05:00:50,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-27 05:00:50,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-27 05:00:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-27 05:00:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-27 05:00:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-27 05:00:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-27 05:00:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-27 05:00:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-27 05:00:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-27 05:00:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-27 05:00:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-27 05:00:50,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-27 05:00:50,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-27 05:00:50,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-27 05:00:50,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-27 05:00:50,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-27 05:00:50,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-27 05:00:50,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-27 05:00:50,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-27 05:00:50,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-27 05:00:50,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-27 05:00:50,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-27 05:00:50,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-27 05:00:50,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-27 05:00:50,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-27 05:00:50,509 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.6590924490066385E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-27 05:00:50,509 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1207502077:83 , srv1466790716:83 , srv450016811:83 , srv748759115:83 , srv810616215:83 , srv1126969502:84 , srv1356708065:84 , srv1378959108:84 , srv138012263:84 , srv1400570999:84 , srv1424921584:84 , srv1543053003:84 , srv156303569:84 , srv1576983427:84 , srv1699973579:84 , srv1731658509:84 , srv1735173038:84 , srv173965216:84 , srv1784247114:84 , srv1790825918:84 , srv1847900061:84 , srv1855427942:84 , srv1861654460:84 , srv1953908190:84 , srv1967344145:84 , srv2031582397:84 , srv2059973197:84 , srv2099602712:84 , srv262498661:84 , srv340945826:84 , srv350370741:84 , srv450607070:84 , srv517399557:84 , srv519098315:84 , srv753180772:84 , srv786539235:84 , srv807795879:84 , srv851007990:84 , srv930897651:84 , srv936731526:84 , srv940594851:84 , srv942559318:84 , srv101190152:85 , srv1134051180:85 , srv1139040454:85 , srv1242674123:85 , srv1260783733:85 , srv1385484922:85 , srv1455821722:85 , srv1478853152:85 , srv1482755053:85 , srv151603779:85 , srv1524574489:85 , srv1545997570:85 , srv1571643727:85 , srv1658949280:85 , srv1690458557:85 , srv1706164386:85 , srv1748513716:85 , srv1751681885:85 , srv1775134971:85 , srv1785325479:85 , srv178755440:85 , srv1792697605:85 , srv1793672086:85 , srv179712365:85 , srv1836847074:85 , srv1893978183:85 , srv2025105754:85 , srv2073132768:85 , srv2099738448:85 , srv2103981846:85 , srv2143230089:85 , srv229508970:85 , srv231356411:85 , srv240232607:85 , srv310579981:85 , srv316083350:85 , srv351510048:85 , srv435903674:85 , srv453080142:85 , srv472940307:85 , srv578905085:85 , srv583939679:85 , srv59814179:85 , srv661107405:85 , srv6707197:85 , srv694106689:85 , srv714574819:85 , srv719845614:85 , srv731342035:85 , srv731737728:85 , srv744308897:85 , srv823487622:85 , srv964075399:85 , srv968011415:85 , srv1493006602:86 , srv1635109997:86 , srv2031331345:86 , srv1293423040:629 }
2024-04-27 05:00:50,532 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=805 (was 805), ProcessCount=180 (was 180), AvailableMemoryMB=1901 (was 2404)
2024-04-27 05:00:50,540 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=805, ProcessCount=180, AvailableMemoryMB=1901
2024-04-27 05:00:50,541 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-04-27 05:00:50,541 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-04-27 05:00:50,548 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1059153642:84 , srv1069605050:84 , srv1101298683:84 , srv1132669320:84 , srv1221990919:84 , srv1493294295:84 , srv1647380460:84 , srv1664166063:84 , srv1761947565:84 , srv182456155:84 , srv1900931883:84 , srv1914410630:84 , srv1968440575:84 , srv2024043827:84 , srv2026091927:84 , srv2030040245:84 , srv2031493436:84 , srv2056461642:84 , srv2130116849:84 , srv213587406:84 , srv249881281:84 , srv278309543:84 , srv278689123:84 , srv290888178:84 , srv483964313:84 , srv533890286:84 , srv557215769:84 , srv732831919:84 , srv953425689:84 , srv1404553331:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-04-27 05:00:50,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-27 05:00:50,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1647380460=7, srv483964313=25, srv2024043827=14, srv2026091927=15, srv533890286=26, srv278309543=22, srv1132669320=3, srv1761947565=9, srv182456155=10, srv1900931883=11, srv1069605050=1, srv249881281=21, srv1664166063=8, srv290888178=24, srv2030040245=16, srv953425689=29, srv2031493436=17, srv1221990919=4, srv1493294295=6, srv213587406=20, srv732831919=28, srv1914410630=12, srv2056461642=18, srv1059153642=0, srv278689123=23, srv557215769=27, srv2130116849=19, srv1101298683=2, srv1404553331=5, srv1968440575=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-27 05:00:50,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-27 05:00:50,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-27 05:00:50,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-27 05:00:50,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-27 05:00:50,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-27 05:00:50,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-27 05:00:50,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-27 05:00:50,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-27 05:00:50,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-27 05:00:50,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-27 05:00:50,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-27 05:00:50,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-04-27 05:00:50,571 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.101523829958378, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-27 05:00:50,571 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912512941537421, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.101523829958378, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-04-27 05:02:50,561 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9326237 different iterations.  Found a solution that moves 1837 regions; Going from a computed imbalance of 0.9912512941537421 to a new imbalance of 4.2876496417733574E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6803703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-27 05:02:50,566 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1059153642:90 , srv1069605050:90 , srv1101298683:90 , srv1132669320:90 , srv1221990919:90 , srv1404553331:90 , srv1493294295:90 , srv1647380460:90 , srv1664166063:90 , srv1761947565:90 , srv182456155:90 , srv1900931883:90 , srv1914410630:90 , srv1968440575:90 , srv2024043827:90 , srv2026091927:90 , srv2030040245:90 , srv2031493436:90 , srv2056461642:90 , srv2130116849:90 , srv213587406:90 , srv249881281:90 , srv278309543:90 , srv278689123:90 , srv290888178:90 , srv483964313:90 , srv533890286:90 , srv557215769:90 , srv732831919:90 , srv953425689:90 }
2024-04-27 05:02:50,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-27 05:02:50,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1647380460=7, srv483964313=25, srv2024043827=14, srv2026091927=15, srv533890286=26, srv278309543=22, srv1132669320=3, srv1761947565=9, srv182456155=10, srv1900931883=11, srv1069605050=1, srv249881281=21, srv1664166063=8, srv290888178=24, srv2030040245=16, srv953425689=29, srv2031493436=17, srv1221990919=4, srv1493294295=6, srv213587406=20, srv732831919=28, srv1914410630=12, srv2056461642=18, srv1059153642=0, srv278689123=23, srv557215769=27, srv2130116849=19, srv1101298683=2, srv1404553331=5, srv1968440575=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-27 05:02:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-27 05:02:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-27 05:02:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-27 05:02:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-27 05:02:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-27 05:02:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-27 05:02:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-27 05:02:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-27 05:02:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-27 05:02:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-27 05:02:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-27 05:02:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-04-27 05:02:50,581 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-27 05:02:50,581 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1059153642:90 , srv1069605050:90 , srv1101298683:90 , srv1132669320:90 , srv1221990919:90 , srv1404553331:90 , srv1493294295:90 , srv1647380460:90 , srv1664166063:90 , srv1761947565:90 , srv182456155:90 , srv1900931883:90 , srv1914410630:90 , srv1968440575:90 , srv2024043827:90 , srv2026091927:90 , srv2030040245:90 , srv2031493436:90 , srv2056461642:90 , srv2130116849:90 , srv213587406:90 , srv249881281:90 , srv278309543:90 , srv278689123:90 , srv290888178:90 , srv483964313:90 , srv533890286:90 , srv557215769:90 , srv732831919:90 , srv953425689:90 }
2024-04-27 05:02:50,594 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 805), ProcessCount=175 (was 180), AvailableMemoryMB=2333 (was 1901) - AvailableMemoryMB LEAK? -