2023-05-08 05:53:37,582 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-05-08 05:53:37,932 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-05-08 05:53:38,051 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-05-08 05:53:38,118 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-08 05:53:38,126 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-08 05:53:38,141 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=291, ProcessCount=173, AvailableMemoryMB=15417
2023-05-08 05:53:38,144 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-08 05:53:38,145 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-08 05:53:38,291 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1017373006:84 , srv1064808794:84 , srv1092665423:84 , srv1135029244:84 , srv1156679995:84 , srv1264209726:84 , srv1269261571:84 , srv1271597418:84 , srv1285274965:84 , srv130382796:84 , srv1308999048:84 , srv1313299693:84 , srv1330289334:84 , srv1342081977:84 , srv1346976464:84 , srv1348575373:84 , srv1431765918:84 , srv1432670476:84 , srv1455509565:84 , srv1481581066:84 , srv1490755224:84 , srv1498361166:84 , srv1513273598:84 , srv1530046768:84 , srv1530410883:84 , srv1530628604:84 , srv153567373:84 , srv1540767649:84 , srv1547425083:84 , srv1553283658:84 , srv1554297916:84 , srv1591498884:84 , srv1591899212:84 , srv1600327707:84 , srv1618773692:84 , srv1642134457:84 , srv1657565417:84 , srv1662475998:84 , srv167126572:84 , srv1675768887:84 , srv1741088584:84 , srv1784423803:84 , srv1807715639:84 , srv1814189817:84 , srv1839179435:84 , srv1846432294:84 , srv1853536218:84 , srv1948544626:84 , srv1963436568:84 , srv1967467085:84 , srv1979871466:84 , srv2014374150:84 , srv2072037173:84 , srv2120387067:84 , srv221053660:84 , srv232227724:84 , srv234591831:84 , srv286865410:84 , srv323509435:84 , srv33933400:84 , srv350293369:84 , srv360757435:84 , srv390762081:84 , srv412662287:84 , srv426616083:84 , srv443619624:84 , srv443681454:84 , srv454473352:84 , srv476247067:84 , srv501430882:84 , srv504030009:84 , srv551727084:84 , srv571701121:84 , srv578487313:84 , srv604883194:84 , srv624964605:84 , srv639264597:84 , srv675410818:84 , srv679555429:84 , srv695753005:84 , srv705636293:84 , srv711644182:84 , srv712699240:84 , srv718613043:84 , srv750056961:84 , srv774678256:84 , srv784712534:84 , srv824494548:84 , srv838555736:84 , srv842357643:84 , srv871783032:84 , srv885505628:84 , srv910764654:84 , srv913232873:84 , srv970667382:84 , srv972784464:84 , srv982488767:84 , srv991875974:84 , srv996644454:84 , srv1997600374:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-05-08 05:53:38,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-08 05:53:38,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv234591831=57, srv1600327707=33, srv2014374150=52, srv774678256=86, srv784712534=87, srv1967467085=49, srv1498361166=21, srv1092665423=2, srv350293369=61, srv1313299693=11, srv221053660=55, srv842357643=90, srv1490755224=20, srv1064808794=1, srv426616083=65, srv1455509565=18, srv232227724=56, srv972784464=96, srv1784423803=41, srv679555429=79, srv443619624=66, srv1135029244=3, srv718613043=84, srv711644182=82, srv1431765918=16, srv390762081=63, srv750056961=85, srv705636293=81, srv2120387067=54, srv323509435=59, srv1839179435=44, srv1948544626=47, srv130382796=9, srv2072037173=53, srv1285274965=8, srv33933400=60, srv443681454=67, srv1591899212=32, srv1308999048=10, srv1017373006=0, srv1807715639=42, srv1814189817=43, srv824494548=88, srv571701121=73, srv1554297916=30, srv153567373=26, srv1642134457=35, srv624964605=76, srv1618773692=34, srv1481581066=19, srv1553283658=29, srv167126572=38, srv578487313=74, srv1264209726=5, srv1530628604=25, srv1547425083=28, srv360757435=62, srv1979871466=50, srv695753005=80, srv551727084=72, srv991875974=98, srv871783032=91, srv1540767649=27, srv1330289334=12, srv970667382=95, srv1346976464=14, srv885505628=92, srv1963436568=48, srv1156679995=4, srv1342081977=13, srv604883194=75, srv1591498884=31, srv1530046768=23, srv712699240=83, srv1675768887=39, srv1853536218=46, srv982488767=97, srv1348575373=15, srv286865410=58, srv1513273598=22, srv476247067=69, srv913232873=94, srv838555736=89, srv1741088584=40, srv412662287=64, srv1657565417=36, srv996644454=99, srv675410818=78, srv639264597=77, srv504030009=71, srv454473352=68, srv1530410883=24, srv1846432294=45, srv1662475998=37, srv910764654=93, srv1269261571=6, srv1997600374=51, srv1432670476=17, srv1271597418=7, srv501430882=70} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-08 05:53:38,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-08 05:53:38,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-08 05:53:38,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-08 05:53:38,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-08 05:53:38,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-08 05:53:38,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-08 05:53:38,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-08 05:53:38,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-08 05:53:38,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-08 05:53:38,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-08 05:53:38,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-08 05:53:38,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-08 05:53:38,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-08 05:53:38,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-08 05:53:38,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-08 05:53:38,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-08 05:53:38,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-08 05:53:38,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-08 05:53:38,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-08 05:53:38,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-08 05:53:38,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-08 05:53:38,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-08 05:53:38,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-08 05:53:38,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-08 05:53:38,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-08 05:53:38,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-08 05:53:38,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-08 05:53:38,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-08 05:53:38,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-08 05:53:38,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-08 05:53:38,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-08 05:53:38,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-08 05:53:38,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-08 05:53:38,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-08 05:53:38,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-08 05:53:38,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-08 05:53:38,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-08 05:53:38,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-08 05:53:38,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-08 05:53:38,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-08 05:53:38,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-08 05:53:38,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-08 05:53:38,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-08 05:53:38,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-08 05:53:38,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-08 05:53:38,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-08 05:53:38,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-08 05:53:38,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-08 05:53:38,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-08 05:53:38,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-08 05:53:38,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-08 05:53:38,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-08 05:53:38,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-08 05:53:38,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-08 05:53:38,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-08 05:53:38,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-08 05:53:38,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-08 05:53:38,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-08 05:53:38,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-08 05:53:38,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-08 05:53:38,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-08 05:53:38,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-08 05:53:38,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-08 05:53:38,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-08 05:53:38,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-08 05:53:38,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-08 05:53:38,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-08 05:53:38,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-08 05:53:38,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-08 05:53:38,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-08 05:53:38,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-08 05:53:38,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-08 05:53:38,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-08 05:53:38,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-08 05:53:38,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-08 05:53:38,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-08 05:53:38,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-08 05:53:38,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-08 05:53:38,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-08 05:53:38,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-08 05:53:38,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-08 05:53:38,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-08 05:53:38,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-08 05:53:38,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-08 05:53:38,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-08 05:53:38,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-08 05:53:38,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-08 05:53:38,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-08 05:53:38,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-08 05:53:38,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-08 05:53:38,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-08 05:53:38,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-08 05:53:38,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-08 05:53:38,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-08 05:53:38,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-08 05:53:38,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-08 05:53:38,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-08 05:53:38,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-08 05:53:38,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-08 05:53:38,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-08 05:53:38,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-08 05:53:38,696 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-08 05:53:38,696 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-08 05:53:38,697 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-08 05:53:39,027 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 567 ms to try 5000 different iterations.  Found a solution that moves 4702 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1795979260333314. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04060890822050261, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057631905990547065, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5224444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04060890822050261, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17491666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24033333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-08 05:53:39,037 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv972784464:79 , srv1675768887:81 , srv639264597:81 , srv167126572:82 , srv982488767:82 , srv1308999048:83 , srv1618773692:83 , srv1979871466:83 , srv390762081:83 , srv1156679995:84 , srv1490755224:84 , srv2014374150:84 , srv232227724:84 , srv578487313:84 , srv711644182:84 , srv824494548:84 , srv1135029244:85 , srv1271597418:85 , srv1346976464:85 , srv1455509565:85 , srv1481581066:85 , srv1498361166:85 , srv1540767649:85 , srv1554297916:85 , srv1591498884:85 , srv1662475998:85 , srv2120387067:85 , srv360757435:85 , srv454473352:85 , srv476247067:85 , srv504030009:85 , srv913232873:85 , srv970667382:85 , srv1092665423:86 , srv1264209726:86 , srv1269261571:86 , srv1285274965:86 , srv1348575373:86 , srv1432670476:86 , srv1600327707:86 , srv1642134457:86 , srv1657565417:86 , srv1784423803:86 , srv1846432294:86 , srv1967467085:86 , srv2072037173:86 , srv350293369:86 , srv412662287:86 , srv426616083:86 , srv443619624:86 , srv443681454:86 , srv679555429:86 , srv695753005:86 , srv712699240:86 , srv718613043:86 , srv750056961:86 , srv842357643:86 , srv871783032:86 , srv910764654:86 , srv130382796:87 , srv1530410883:87 , srv1553283658:87 , srv1741088584:87 , srv1807715639:87 , srv1948544626:87 , srv221053660:87 , srv501430882:87 , srv571701121:87 , srv624964605:87 , srv774678256:87 , srv838555736:87 , srv885505628:87 , srv1064808794:88 , srv1431765918:88 , srv1513273598:88 , srv1530628604:88 , srv1547425083:88 , srv1814189817:88 , srv1853536218:88 , srv33933400:88 , srv675410818:88 , srv705636293:88 , srv996644454:88 , srv1313299693:89 , srv1342081977:89 , srv1839179435:89 , srv234591831:89 , srv286865410:89 , srv551727084:89 , srv1017373006:90 , srv1330289334:90 , srv1530046768:90 , srv153567373:90 , srv1591899212:90 , srv323509435:90 , srv604883194:90 , srv991875974:90 , srv1963436568:94 , srv784712534:96 , srv1997600374:451 }
2023-05-08 05:53:39,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-08 05:53:39,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv234591831=57, srv1600327707=33, srv2014374150=52, srv774678256=86, srv784712534=87, srv1967467085=49, srv1498361166=21, srv1092665423=2, srv350293369=61, srv1313299693=11, srv221053660=55, srv842357643=90, srv1490755224=20, srv1064808794=1, srv426616083=65, srv1455509565=18, srv232227724=56, srv972784464=96, srv1784423803=41, srv679555429=79, srv443619624=66, srv1135029244=3, srv718613043=84, srv711644182=82, srv1431765918=16, srv390762081=63, srv750056961=85, srv705636293=81, srv2120387067=54, srv323509435=59, srv1839179435=44, srv1948544626=47, srv130382796=9, srv2072037173=53, srv1285274965=8, srv33933400=60, srv443681454=67, srv1591899212=32, srv1308999048=10, srv1017373006=0, srv1807715639=42, srv1814189817=43, srv824494548=88, srv571701121=73, srv1554297916=30, srv153567373=26, srv1642134457=35, srv624964605=76, srv1618773692=34, srv1481581066=19, srv1553283658=29, srv167126572=38, srv578487313=74, srv1264209726=5, srv1530628604=25, srv1547425083=28, srv360757435=62, srv1979871466=50, srv695753005=80, srv551727084=72, srv991875974=98, srv871783032=91, srv1540767649=27, srv1330289334=12, srv970667382=95, srv1346976464=14, srv885505628=92, srv1963436568=48, srv1156679995=4, srv1342081977=13, srv604883194=75, srv1591498884=31, srv1530046768=23, srv712699240=83, srv1675768887=39, srv1853536218=46, srv982488767=97, srv1348575373=15, srv286865410=58, srv1513273598=22, srv476247067=69, srv913232873=94, srv838555736=89, srv1741088584=40, srv412662287=64, srv1657565417=36, srv996644454=99, srv675410818=78, srv639264597=77, srv504030009=71, srv454473352=68, srv1530410883=24, srv1846432294=45, srv1662475998=37, srv910764654=93, srv1269261571=6, srv1997600374=51, srv1432670476=17, srv1271597418=7, srv501430882=70} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-08 05:53:39,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-08 05:53:39,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-08 05:53:39,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-08 05:53:39,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-08 05:53:39,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-08 05:53:39,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-08 05:53:39,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-08 05:53:39,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-08 05:53:39,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-08 05:53:39,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-08 05:53:39,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-08 05:53:39,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-08 05:53:39,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-08 05:53:39,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-08 05:53:39,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-08 05:53:39,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-08 05:53:39,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-08 05:53:39,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-08 05:53:39,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-08 05:53:39,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-08 05:53:39,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-08 05:53:39,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-08 05:53:39,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-08 05:53:39,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-08 05:53:39,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-08 05:53:39,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-08 05:53:39,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-08 05:53:39,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-08 05:53:39,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-08 05:53:39,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-08 05:53:39,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-08 05:53:39,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-08 05:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-08 05:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-08 05:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-08 05:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-08 05:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-08 05:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-08 05:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-08 05:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-08 05:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-08 05:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-08 05:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-08 05:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-08 05:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-08 05:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-08 05:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-08 05:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-08 05:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-08 05:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-08 05:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-08 05:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-08 05:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-08 05:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-08 05:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-08 05:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-08 05:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-08 05:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-08 05:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-08 05:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-08 05:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-08 05:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-08 05:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-08 05:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-08 05:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-08 05:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-08 05:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-08 05:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-08 05:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-08 05:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-08 05:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-08 05:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-08 05:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-08 05:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-08 05:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-08 05:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-08 05:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-08 05:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-08 05:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-08 05:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-08 05:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-08 05:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-08 05:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-08 05:53:39,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-08 05:53:39,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-08 05:53:39,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-08 05:53:39,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-08 05:53:39,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-08 05:53:39,214 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04060890822050261, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057631905990547065, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04060890822050261, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17491666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24033333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-08 05:53:39,214 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-08 05:53:39,215 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17956500192563032, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04060890822050261, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057631905990547065, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04060890822050261, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17491666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24033333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-08 05:53:39,570 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 479 ms to try 5000 different iterations.  Found a solution that moves 2646 regions; Going from a computed imbalance of 0.17956500192563032 to a new imbalance of 0.014964311914579568. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02441024032853568); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03581345858664238, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.294, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02441024032853568); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-08 05:53:39,584 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv784712534:77 , srv1963436568:79 , srv1530046768:81 , srv1591899212:81 , srv286865410:81 , srv323509435:81 , srv604883194:81 , srv1064808794:82 , srv1313299693:82 , srv1431765918:82 , srv1513273598:82 , srv1553283658:82 , srv1814189817:82 , srv33933400:82 , srv705636293:82 , srv1342081977:83 , srv1530628604:83 , srv1839179435:83 , srv234591831:83 , srv350293369:83 , srv675410818:83 , srv679555429:83 , srv774678256:83 , srv996644454:83 , srv1017373006:84 , srv1269261571:84 , srv1330289334:84 , srv1547425083:84 , srv1642134457:84 , srv221053660:84 , srv913232873:84 , srv991875974:84 , srv1264209726:85 , srv130382796:85 , srv1348575373:85 , srv1432670476:85 , srv153567373:85 , srv1853536218:85 , srv1948544626:85 , srv232227724:85 , srv412662287:85 , srv443681454:85 , srv551727084:85 , srv571701121:85 , srv624964605:85 , srv712699240:85 , srv838555736:85 , srv842357643:85 , srv1092665423:86 , srv1135029244:86 , srv1346976464:86 , srv1455509565:86 , srv1530410883:86 , srv1600327707:86 , srv1657565417:86 , srv1741088584:86 , srv1784423803:86 , srv1807715639:86 , srv1846432294:86 , srv2014374150:86 , srv2072037173:86 , srv2120387067:86 , srv504030009:86 , srv695753005:86 , srv750056961:86 , srv824494548:86 , srv910764654:86 , srv1156679995:87 , srv1271597418:87 , srv1481581066:87 , srv1554297916:87 , srv1591498884:87 , srv1662475998:87 , srv1979871466:87 , srv426616083:87 , srv443619624:87 , srv501430882:87 , srv578487313:87 , srv885505628:87 , srv1285274965:88 , srv1308999048:88 , srv1498361166:88 , srv1540767649:88 , srv167126572:88 , srv1967467085:88 , srv476247067:88 , srv1490755224:89 , srv360757435:89 , srv390762081:89 , srv711644182:89 , srv718613043:89 , srv871783032:89 , srv970667382:89 , srv982488767:89 , srv454473352:90 , srv1618773692:91 , srv1675768887:91 , srv639264597:92 , srv972784464:94 , srv1997600374:540 }
2023-05-08 05:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-08 05:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv234591831=57, srv1600327707=33, srv2014374150=52, srv774678256=86, srv784712534=87, srv1967467085=49, srv1498361166=21, srv1092665423=2, srv350293369=61, srv1313299693=11, srv221053660=55, srv842357643=90, srv1490755224=20, srv1064808794=1, srv426616083=65, srv1455509565=18, srv232227724=56, srv972784464=96, srv1784423803=41, srv679555429=79, srv443619624=66, srv1135029244=3, srv718613043=84, srv711644182=82, srv1431765918=16, srv390762081=63, srv750056961=85, srv705636293=81, srv2120387067=54, srv323509435=59, srv1839179435=44, srv1948544626=47, srv130382796=9, srv2072037173=53, srv1285274965=8, srv33933400=60, srv443681454=67, srv1591899212=32, srv1308999048=10, srv1017373006=0, srv1807715639=42, srv1814189817=43, srv824494548=88, srv571701121=73, srv1554297916=30, srv153567373=26, srv1642134457=35, srv624964605=76, srv1618773692=34, srv1481581066=19, srv1553283658=29, srv167126572=38, srv578487313=74, srv1264209726=5, srv1530628604=25, srv1547425083=28, srv360757435=62, srv1979871466=50, srv695753005=80, srv551727084=72, srv991875974=98, srv871783032=91, srv1540767649=27, srv1330289334=12, srv970667382=95, srv1346976464=14, srv885505628=92, srv1963436568=48, srv1156679995=4, srv1342081977=13, srv604883194=75, srv1591498884=31, srv1530046768=23, srv712699240=83, srv1675768887=39, srv1853536218=46, srv982488767=97, srv1348575373=15, srv286865410=58, srv1513273598=22, srv476247067=69, srv913232873=94, srv838555736=89, srv1741088584=40, srv412662287=64, srv1657565417=36, srv996644454=99, srv675410818=78, srv639264597=77, srv504030009=71, srv454473352=68, srv1530410883=24, srv1846432294=45, srv1662475998=37, srv910764654=93, srv1269261571=6, srv1997600374=51, srv1432670476=17, srv1271597418=7, srv501430882=70} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-08 05:53:39,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-08 05:53:39,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-08 05:53:39,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-08 05:53:39,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-08 05:53:39,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-08 05:53:39,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-08 05:53:39,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-08 05:53:39,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-08 05:53:39,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-08 05:53:39,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-08 05:53:39,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-08 05:53:39,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-08 05:53:39,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-08 05:53:39,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-08 05:53:39,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-08 05:53:39,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-08 05:53:39,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-08 05:53:39,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-08 05:53:39,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-08 05:53:39,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-08 05:53:39,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-08 05:53:39,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-08 05:53:39,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-08 05:53:39,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-08 05:53:39,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-08 05:53:39,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-08 05:53:39,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-08 05:53:39,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-08 05:53:39,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-08 05:53:39,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-08 05:53:39,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-08 05:53:39,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-08 05:53:39,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-08 05:53:39,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-08 05:53:39,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-08 05:53:39,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-08 05:53:39,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-08 05:53:39,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-08 05:53:39,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-08 05:53:39,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-08 05:53:39,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-08 05:53:39,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-08 05:53:39,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-08 05:53:39,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-08 05:53:39,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-08 05:53:39,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-08 05:53:39,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-08 05:53:39,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-08 05:53:39,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-08 05:53:39,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-08 05:53:39,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-08 05:53:39,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-08 05:53:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-08 05:53:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-08 05:53:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-08 05:53:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-08 05:53:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-08 05:53:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-08 05:53:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-08 05:53:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-08 05:53:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-08 05:53:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-08 05:53:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-08 05:53:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-08 05:53:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-08 05:53:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-08 05:53:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-08 05:53:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-08 05:53:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-08 05:53:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-08 05:53:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-08 05:53:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-08 05:53:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-08 05:53:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-08 05:53:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-08 05:53:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-08 05:53:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-08 05:53:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-08 05:53:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-08 05:53:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-08 05:53:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-08 05:53:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-08 05:53:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-08 05:53:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-08 05:53:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-08 05:53:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-08 05:53:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-08 05:53:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-08 05:53:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-08 05:53:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-08 05:53:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-08 05:53:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-08 05:53:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-08 05:53:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-08 05:53:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-08 05:53:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-08 05:53:39,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-08 05:53:39,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-08 05:53:39,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-08 05:53:39,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-08 05:53:39,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-08 05:53:39,771 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02441024032853568); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03581345858664238, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02441024032853568); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-08 05:53:39,771 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-08 05:53:39,771 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014945784226579351, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02441024032853568); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03581345858664238, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02441024032853568); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-08 05:53:40,109 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 469 ms to try 5000 different iterations.  Found a solution that moves 993 regions; Going from a computed imbalance of 0.014945784226579351 to a new imbalance of 9.30362655889604E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007235376090120124); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009063933040070526); MoveCostFunction : (multiplier=7.0, imbalance=0.11033333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007235376090120124); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-08 05:53:40,112 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1330289334:82 , srv454473352:82 , srv871783032:82 , srv153567373:83 , srv1540767649:83 , srv476247067:83 , srv501430882:83 , srv551727084:83 , srv718613043:83 , srv885505628:83 , srv1017373006:84 , srv1285274965:84 , srv1490755224:84 , srv1498361166:84 , srv1530410883:84 , srv1741088584:84 , srv1846432294:84 , srv1967467085:84 , srv390762081:84 , srv443619624:84 , srv639264597:84 , srv784712534:84 , srv970667382:84 , srv972784464:84 , srv1092665423:85 , srv1264209726:85 , srv1342081977:85 , srv1481581066:85 , srv1547425083:85 , srv1554297916:85 , srv1591498884:85 , srv1600327707:85 , srv1618773692:85 , srv1662475998:85 , srv1784423803:85 , srv1948544626:85 , srv1963436568:85 , srv234591831:85 , srv360757435:85 , srv426616083:85 , srv695753005:85 , srv711644182:85 , srv750056961:85 , srv838555736:85 , srv910764654:85 , srv991875974:85 , srv1064808794:86 , srv1135029244:86 , srv1271597418:86 , srv130382796:86 , srv1313299693:86 , srv1348575373:86 , srv1431765918:86 , srv1432670476:86 , srv1530046768:86 , srv1530628604:86 , srv1591899212:86 , srv1642134457:86 , srv1657565417:86 , srv1675768887:86 , srv1807715639:86 , srv1839179435:86 , srv1853536218:86 , srv1979871466:86 , srv2072037173:86 , srv221053660:86 , srv323509435:86 , srv412662287:86 , srv443681454:86 , srv571701121:86 , srv578487313:86 , srv604883194:86 , srv624964605:86 , srv824494548:86 , srv842357643:86 , srv996644454:86 , srv1156679995:87 , srv1346976464:87 , srv1455509565:87 , srv1513273598:87 , srv2014374150:87 , srv2120387067:87 , srv286865410:87 , srv33933400:87 , srv504030009:87 , srv675410818:87 , srv712699240:87 , srv982488767:87 , srv1269261571:88 , srv1308999048:88 , srv1553283658:88 , srv167126572:88 , srv232227724:88 , srv679555429:88 , srv705636293:88 , srv1814189817:89 , srv350293369:89 , srv774678256:89 , srv913232873:89 , srv1997600374:531 }
2023-05-08 05:53:40,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-08 05:53:40,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv234591831=57, srv1600327707=33, srv2014374150=52, srv774678256=86, srv784712534=87, srv1967467085=49, srv1498361166=21, srv1092665423=2, srv350293369=61, srv1313299693=11, srv221053660=55, srv842357643=90, srv1490755224=20, srv1064808794=1, srv426616083=65, srv1455509565=18, srv232227724=56, srv972784464=96, srv1784423803=41, srv679555429=79, srv443619624=66, srv1135029244=3, srv718613043=84, srv711644182=82, srv1431765918=16, srv390762081=63, srv750056961=85, srv705636293=81, srv2120387067=54, srv323509435=59, srv1839179435=44, srv1948544626=47, srv130382796=9, srv2072037173=53, srv1285274965=8, srv33933400=60, srv443681454=67, srv1591899212=32, srv1308999048=10, srv1017373006=0, srv1807715639=42, srv1814189817=43, srv824494548=88, srv571701121=73, srv1554297916=30, srv153567373=26, srv1642134457=35, srv624964605=76, srv1618773692=34, srv1481581066=19, srv1553283658=29, srv167126572=38, srv578487313=74, srv1264209726=5, srv1530628604=25, srv1547425083=28, srv360757435=62, srv1979871466=50, srv695753005=80, srv551727084=72, srv991875974=98, srv871783032=91, srv1540767649=27, srv1330289334=12, srv970667382=95, srv1346976464=14, srv885505628=92, srv1963436568=48, srv1156679995=4, srv1342081977=13, srv604883194=75, srv1591498884=31, srv1530046768=23, srv712699240=83, srv1675768887=39, srv1853536218=46, srv982488767=97, srv1348575373=15, srv286865410=58, srv1513273598=22, srv476247067=69, srv913232873=94, srv838555736=89, srv1741088584=40, srv412662287=64, srv1657565417=36, srv996644454=99, srv675410818=78, srv639264597=77, srv504030009=71, srv454473352=68, srv1530410883=24, srv1846432294=45, srv1662475998=37, srv910764654=93, srv1269261571=6, srv1997600374=51, srv1432670476=17, srv1271597418=7, srv501430882=70} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-08 05:53:40,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-08 05:53:40,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-08 05:53:40,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-08 05:53:40,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-08 05:53:40,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-08 05:53:40,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-08 05:53:40,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-08 05:53:40,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-08 05:53:40,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-08 05:53:40,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-08 05:53:40,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-08 05:53:40,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-08 05:53:40,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-08 05:53:40,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-08 05:53:40,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-08 05:53:40,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-08 05:53:40,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-08 05:53:40,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-08 05:53:40,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-08 05:53:40,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-08 05:53:40,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-08 05:53:40,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-08 05:53:40,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-08 05:53:40,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-08 05:53:40,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-08 05:53:40,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-08 05:53:40,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-08 05:53:40,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-08 05:53:40,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-08 05:53:40,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-08 05:53:40,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-08 05:53:40,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-08 05:53:40,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-08 05:53:40,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-08 05:53:40,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-08 05:53:40,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-08 05:53:40,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-08 05:53:40,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-08 05:53:40,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-08 05:53:40,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-08 05:53:40,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-08 05:53:40,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-08 05:53:40,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-08 05:53:40,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-08 05:53:40,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-08 05:53:40,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-08 05:53:40,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-08 05:53:40,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-08 05:53:40,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-08 05:53:40,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-08 05:53:40,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-08 05:53:40,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-08 05:53:40,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-08 05:53:40,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-08 05:53:40,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-08 05:53:40,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-08 05:53:40,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-08 05:53:40,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-08 05:53:40,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-08 05:53:40,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-08 05:53:40,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-08 05:53:40,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-08 05:53:40,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-08 05:53:40,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-08 05:53:40,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-08 05:53:40,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-08 05:53:40,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-08 05:53:40,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-08 05:53:40,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-08 05:53:40,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-08 05:53:40,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-08 05:53:40,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-08 05:53:40,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-08 05:53:40,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-08 05:53:40,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-08 05:53:40,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-08 05:53:40,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-08 05:53:40,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-08 05:53:40,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-08 05:53:40,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-08 05:53:40,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-08 05:53:40,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-08 05:53:40,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-08 05:53:40,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-08 05:53:40,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-08 05:53:40,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-08 05:53:40,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-08 05:53:40,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-08 05:53:40,270 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007235376090120124); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009063933040070526); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007235376090120124); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-08 05:53:40,270 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-08 05:53:40,270 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.23409521277278E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007235376090120124); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009063933040070526); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007235376090120124); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-08 05:53:40,599 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 435 ms to try 5000 different iterations.  Found a solution that moves 281 regions; Going from a computed imbalance of 9.23409521277278E-4 to a new imbalance of 2.2749776487592766E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.03122222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-08 05:53:40,600 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1308999048:83 , srv1618773692:83 , srv1807715639:83 , srv1814189817:83 , srv1963436568:83 , srv360757435:83 , srv774678256:83 , srv970667382:83 , srv991875974:83 , srv1135029244:84 , srv1156679995:84 , srv1269261571:84 , srv1271597418:84 , srv1285274965:84 , srv130382796:84 , srv1346976464:84 , srv1455509565:84 , srv1481581066:84 , srv153567373:84 , srv1591498884:84 , srv1591899212:84 , srv1657565417:84 , srv167126572:84 , srv1675768887:84 , srv1853536218:84 , srv2072037173:84 , srv2120387067:84 , srv221053660:84 , srv234591831:84 , srv350293369:84 , srv426616083:84 , srv504030009:84 , srv571701121:84 , srv624964605:84 , srv705636293:84 , srv711644182:84 , srv712699240:84 , srv718613043:84 , srv871783032:84 , srv913232873:84 , srv982488767:84 , srv1017373006:85 , srv1092665423:85 , srv1313299693:85 , srv1342081977:85 , srv1348575373:85 , srv1431765918:85 , srv1490755224:85 , srv1498361166:85 , srv1513273598:85 , srv1530046768:85 , srv1530410883:85 , srv1530628604:85 , srv1547425083:85 , srv1553283658:85 , srv1554297916:85 , srv1600327707:85 , srv1662475998:85 , srv1741088584:85 , srv1784423803:85 , srv1839179435:85 , srv1948544626:85 , srv1967467085:85 , srv2014374150:85 , srv232227724:85 , srv286865410:85 , srv323509435:85 , srv412662287:85 , srv443619624:85 , srv443681454:85 , srv501430882:85 , srv551727084:85 , srv578487313:85 , srv604883194:85 , srv639264597:85 , srv675410818:85 , srv679555429:85 , srv695753005:85 , srv750056961:85 , srv784712534:85 , srv842357643:85 , srv885505628:85 , srv910764654:85 , srv972784464:85 , srv996644454:85 , srv1064808794:86 , srv1264209726:86 , srv1330289334:86 , srv1432670476:86 , srv1540767649:86 , srv1642134457:86 , srv1846432294:86 , srv1979871466:86 , srv33933400:86 , srv390762081:86 , srv454473352:86 , srv824494548:86 , srv838555736:86 , srv476247067:87 , srv1997600374:620 }
2023-05-08 05:53:40,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-08 05:53:40,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv234591831=57, srv1600327707=33, srv2014374150=52, srv774678256=86, srv784712534=87, srv1967467085=49, srv1498361166=21, srv1092665423=2, srv350293369=61, srv1313299693=11, srv221053660=55, srv842357643=90, srv1490755224=20, srv1064808794=1, srv426616083=65, srv1455509565=18, srv232227724=56, srv972784464=96, srv1784423803=41, srv679555429=79, srv443619624=66, srv1135029244=3, srv718613043=84, srv711644182=82, srv1431765918=16, srv390762081=63, srv750056961=85, srv705636293=81, srv2120387067=54, srv323509435=59, srv1839179435=44, srv1948544626=47, srv130382796=9, srv2072037173=53, srv1285274965=8, srv33933400=60, srv443681454=67, srv1591899212=32, srv1308999048=10, srv1017373006=0, srv1807715639=42, srv1814189817=43, srv824494548=88, srv571701121=73, srv1554297916=30, srv153567373=26, srv1642134457=35, srv624964605=76, srv1618773692=34, srv1481581066=19, srv1553283658=29, srv167126572=38, srv578487313=74, srv1264209726=5, srv1530628604=25, srv1547425083=28, srv360757435=62, srv1979871466=50, srv695753005=80, srv551727084=72, srv991875974=98, srv871783032=91, srv1540767649=27, srv1330289334=12, srv970667382=95, srv1346976464=14, srv885505628=92, srv1963436568=48, srv1156679995=4, srv1342081977=13, srv604883194=75, srv1591498884=31, srv1530046768=23, srv712699240=83, srv1675768887=39, srv1853536218=46, srv982488767=97, srv1348575373=15, srv286865410=58, srv1513273598=22, srv476247067=69, srv913232873=94, srv838555736=89, srv1741088584=40, srv412662287=64, srv1657565417=36, srv996644454=99, srv675410818=78, srv639264597=77, srv504030009=71, srv454473352=68, srv1530410883=24, srv1846432294=45, srv1662475998=37, srv910764654=93, srv1269261571=6, srv1997600374=51, srv1432670476=17, srv1271597418=7, srv501430882=70} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-08 05:53:40,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-08 05:53:40,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-08 05:53:40,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-08 05:53:40,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-08 05:53:40,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-08 05:53:40,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-08 05:53:40,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-08 05:53:40,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-08 05:53:40,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-08 05:53:40,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-08 05:53:40,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-08 05:53:40,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-08 05:53:40,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-08 05:53:40,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-08 05:53:40,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-08 05:53:40,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-08 05:53:40,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-08 05:53:40,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-08 05:53:40,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-08 05:53:40,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-08 05:53:40,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-08 05:53:40,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-08 05:53:40,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-08 05:53:40,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-08 05:53:40,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-08 05:53:40,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-08 05:53:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-08 05:53:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-08 05:53:40,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-08 05:53:40,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-08 05:53:40,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-08 05:53:40,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-08 05:53:40,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-08 05:53:40,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-08 05:53:40,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-08 05:53:40,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-08 05:53:40,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-08 05:53:40,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-08 05:53:40,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-08 05:53:40,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-08 05:53:40,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-08 05:53:40,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-08 05:53:40,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-08 05:53:40,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-08 05:53:40,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-08 05:53:40,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-08 05:53:40,775 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=2.078217243314805E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-08 05:53:40,776 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1308999048:83 , srv1618773692:83 , srv1807715639:83 , srv1814189817:83 , srv1963436568:83 , srv360757435:83 , srv774678256:83 , srv970667382:83 , srv991875974:83 , srv1135029244:84 , srv1156679995:84 , srv1269261571:84 , srv1271597418:84 , srv1285274965:84 , srv130382796:84 , srv1346976464:84 , srv1455509565:84 , srv1481581066:84 , srv153567373:84 , srv1591498884:84 , srv1591899212:84 , srv1657565417:84 , srv167126572:84 , srv1675768887:84 , srv1853536218:84 , srv2072037173:84 , srv2120387067:84 , srv221053660:84 , srv234591831:84 , srv350293369:84 , srv426616083:84 , srv504030009:84 , srv571701121:84 , srv624964605:84 , srv705636293:84 , srv711644182:84 , srv712699240:84 , srv718613043:84 , srv871783032:84 , srv913232873:84 , srv982488767:84 , srv1017373006:85 , srv1092665423:85 , srv1313299693:85 , srv1342081977:85 , srv1348575373:85 , srv1431765918:85 , srv1490755224:85 , srv1498361166:85 , srv1513273598:85 , srv1530046768:85 , srv1530410883:85 , srv1530628604:85 , srv1547425083:85 , srv1553283658:85 , srv1554297916:85 , srv1600327707:85 , srv1662475998:85 , srv1741088584:85 , srv1784423803:85 , srv1839179435:85 , srv1948544626:85 , srv1967467085:85 , srv2014374150:85 , srv232227724:85 , srv286865410:85 , srv323509435:85 , srv412662287:85 , srv443619624:85 , srv443681454:85 , srv501430882:85 , srv551727084:85 , srv578487313:85 , srv604883194:85 , srv639264597:85 , srv675410818:85 , srv679555429:85 , srv695753005:85 , srv750056961:85 , srv784712534:85 , srv842357643:85 , srv885505628:85 , srv910764654:85 , srv972784464:85 , srv996644454:85 , srv1064808794:86 , srv1264209726:86 , srv1330289334:86 , srv1432670476:86 , srv1540767649:86 , srv1642134457:86 , srv1846432294:86 , srv1979871466:86 , srv33933400:86 , srv390762081:86 , srv454473352:86 , srv824494548:86 , srv838555736:86 , srv476247067:87 , srv1997600374:620 }
2023-05-08 05:53:40,808 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=315 (was 291) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 173), AvailableMemoryMB=14628 (was 15417)
2023-05-08 05:53:40,819 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=315, ProcessCount=173, AvailableMemoryMB=14628
2023-05-08 05:53:40,820 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-08 05:53:40,820 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-05-08 05:53:40,831 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1031858433:84 , srv1121335728:84 , srv1191168916:84 , srv1191567866:84 , srv1217377054:84 , srv125129723:84 , srv1265016369:84 , srv1325733874:84 , srv1422532539:84 , srv1680591591:84 , srv1729502423:84 , srv1794431738:84 , srv1794975382:84 , srv183584346:84 , srv2000678289:84 , srv2101552769:84 , srv255480464:84 , srv258105019:84 , srv292161397:84 , srv419526318:84 , srv431789252:84 , srv504233893:84 , srv619137056:84 , srv733062942:84 , srv746503386:84 , srv766013152:84 , srv770604418:84 , srv838380130:84 , srv988380412:84 , srv1383972634:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-05-08 05:53:40,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-08 05:53:40,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1422532539=9, srv1794975382=13, srv1383972634=8, srv1325733874=7, srv2000678289=15, srv1031858433=0, srv419526318=20, srv1680591591=10, srv255480464=17, srv1729502423=11, srv2101552769=16, srv431789252=21, srv258105019=18, srv1191168916=2, srv1121335728=1, srv1217377054=4, srv988380412=29, srv766013152=26, srv838380130=28, srv125129723=5, srv733062942=24, srv1794431738=12, srv183584346=14, srv1191567866=3, srv619137056=23, srv1265016369=6, srv746503386=25, srv770604418=27, srv292161397=19, srv504233893=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-08 05:53:40,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-08 05:53:40,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-08 05:53:40,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-08 05:53:40,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-08 05:53:40,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-08 05:53:40,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-08 05:53:40,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-08 05:53:40,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-08 05:53:40,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-08 05:53:40,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-08 05:53:40,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-08 05:53:40,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-08 05:53:40,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-08 05:53:40,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-08 05:53:40,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-08 05:53:40,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-08 05:53:40,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-08 05:53:40,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-08 05:53:40,870 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0667079603749312, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-08 05:53:40,871 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912403237869207, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0667079603749312, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-05-08 05:55:40,852 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6411502 different iterations.  Found a solution that moves 1842 regions; Going from a computed imbalance of 0.9912403237869207 to a new imbalance of 4.2993198912066E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6822222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-08 05:55:40,859 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1031858433:90 , srv1121335728:90 , srv1191168916:90 , srv1191567866:90 , srv1217377054:90 , srv125129723:90 , srv1265016369:90 , srv1325733874:90 , srv1383972634:90 , srv1422532539:90 , srv1680591591:90 , srv1729502423:90 , srv1794431738:90 , srv1794975382:90 , srv183584346:90 , srv2000678289:90 , srv2101552769:90 , srv255480464:90 , srv258105019:90 , srv292161397:90 , srv419526318:90 , srv431789252:90 , srv504233893:90 , srv619137056:90 , srv733062942:90 , srv746503386:90 , srv766013152:90 , srv770604418:90 , srv838380130:90 , srv988380412:90 }
2023-05-08 05:55:40,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-08 05:55:40,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1422532539=9, srv1794975382=13, srv1383972634=8, srv1325733874=7, srv2000678289=15, srv1031858433=0, srv419526318=20, srv1680591591=10, srv255480464=17, srv1729502423=11, srv2101552769=16, srv431789252=21, srv258105019=18, srv1191168916=2, srv1121335728=1, srv1217377054=4, srv988380412=29, srv766013152=26, srv838380130=28, srv125129723=5, srv733062942=24, srv1794431738=12, srv183584346=14, srv1191567866=3, srv619137056=23, srv1265016369=6, srv746503386=25, srv770604418=27, srv292161397=19, srv504233893=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-08 05:55:40,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-08 05:55:40,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-08 05:55:40,877 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-08 05:55:40,877 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1031858433:90 , srv1121335728:90 , srv1191168916:90 , srv1191567866:90 , srv1217377054:90 , srv125129723:90 , srv1265016369:90 , srv1325733874:90 , srv1383972634:90 , srv1422532539:90 , srv1680591591:90 , srv1729502423:90 , srv1794431738:90 , srv1794975382:90 , srv183584346:90 , srv2000678289:90 , srv2101552769:90 , srv255480464:90 , srv258105019:90 , srv292161397:90 , srv419526318:90 , srv431789252:90 , srv504233893:90 , srv619137056:90 , srv733062942:90 , srv746503386:90 , srv766013152:90 , srv770604418:90 , srv838380130:90 , srv988380412:90 }
2023-05-08 05:55:40,890 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=229 (was 315), ProcessCount=169 (was 173), AvailableMemoryMB=14498 (was 14628)