2023-03-27 13:54:45,333 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-27 13:54:45,796 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-27 13:54:45,978 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-27 13:54:46,052 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-27 13:54:46,061 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-27 13:54:46,080 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=1320, ProcessCount=207, AvailableMemoryMB=9617
2023-03-27 13:54:46,083 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-27 13:54:46,084 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-27 13:54:46,316 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1043919302:84 , srv1049980706:84 , srv1069272992:84 , srv111102991:84 , srv1119284339:84 , srv1123393060:84 , srv1156221557:84 , srv1173680790:84 , srv1185490534:84 , srv1187158199:84 , srv121250919:84 , srv1230855695:84 , srv1232312247:84 , srv123296776:84 , srv1284514682:84 , srv131977905:84 , srv1329886908:84 , srv1330482500:84 , srv1340684737:84 , srv136130081:84 , srv1423697713:84 , srv1440285310:84 , srv1519941666:84 , srv1554293361:84 , srv1573441745:84 , srv1615845115:84 , srv1633228380:84 , srv1634009801:84 , srv164020636:84 , srv1654262190:84 , srv1662367272:84 , srv1681018455:84 , srv1702124911:84 , srv1716775237:84 , srv1727162507:84 , srv1727710920:84 , srv173683906:84 , srv1759292977:84 , srv1771182623:84 , srv1779218622:84 , srv1801066186:84 , srv1817133068:84 , srv1818249470:84 , srv1894769102:84 , srv1905380746:84 , srv1917311342:84 , srv1948567610:84 , srv1955910012:84 , srv1964488747:84 , srv1972251588:84 , srv1990676088:84 , srv2079630175:84 , srv2084897494:84 , srv2086110549:84 , srv211562863:84 , srv2118559680:84 , srv2134282624:84 , srv2147236586:84 , srv223295684:84 , srv235902957:84 , srv259713415:84 , srv302430793:84 , srv31654380:84 , srv316896813:84 , srv33442412:84 , srv355245942:84 , srv373911620:84 , srv386148969:84 , srv393151695:84 , srv395768122:84 , srv399853691:84 , srv431093254:84 , srv451288569:84 , srv466382273:84 , srv484120592:84 , srv536238880:84 , srv553289064:84 , srv56615975:84 , srv572238852:84 , srv621850103:84 , srv638864615:84 , srv675822337:84 , srv697155186:84 , srv711119170:84 , srv714503247:84 , srv74091037:84 , srv749418874:84 , srv770623518:84 , srv785490953:84 , srv798797433:84 , srv819507800:84 , srv825943573:84 , srv829130356:84 , srv841133863:84 , srv851447204:84 , srv855541105:84 , srv943913165:84 , srv976521083:84 , srv9809780:84 , srv2073649795:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-27 13:54:46,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 13:54:46,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv749418874=87, srv819507800=91, srv1948567610=46, srv302430793=62, srv2134282624=57, srv2118559680=56, srv1615845115=25, srv798797433=90, srv829130356=93, srv1554293361=23, srv393151695=69, srv638864615=81, srv855541105=96, srv2073649795=51, srv74091037=86, srv825943573=92, srv572238852=79, srv841133863=94, srv1702124911=32, srv1654262190=29, srv1801066186=40, srv121250919=10, srv2147236586=58, srv1955910012=47, srv123296776=13, srv1964488747=48, srv553289064=77, srv770623518=88, srv1069272992=2, srv386148969=68, srv1173680790=7, srv395768122=70, srv1727162507=34, srv697155186=83, srv211562863=55, srv1573441745=24, srv1681018455=31, srv373911620=67, srv111102991=3, srv1329886908=16, srv399853691=71, srv943913165=97, srv466382273=74, srv1817133068=41, srv484120592=75, srv1634009801=27, srv714503247=85, srv1662367272=30, srv1771182623=38, srv223295684=59, srv851447204=95, srv1230855695=11, srv1119284339=4, srv1894769102=43, srv173683906=36, srv1917311342=45, srv2079630175=52, srv56615975=78, srv1905380746=44, srv355245942=66, srv1818249470=42, srv131977905=15, srv711119170=84, srv2086110549=54, srv1716775237=33, srv536238880=76, srv1123393060=5, srv1049980706=1, srv1423697713=20, srv431093254=72, srv1043919302=0, srv1972251588=49, srv1759292977=37, srv675822337=82, srv259713415=61, srv2084897494=53, srv976521083=98, srv1185490534=8, srv621850103=80, srv164020636=28, srv31654380=63, srv1779218622=39, srv1156221557=6, srv785490953=89, srv235902957=60, srv136130081=19, srv33442412=65, srv1284514682=14, srv9809780=99, srv1440285310=21, srv1633228380=26, srv451288569=73, srv1232312247=12, srv1340684737=18, srv1990676088=50, srv316896813=64, srv1519941666=22, srv1187158199=9, srv1727710920=35, srv1330482500=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 13:54:46,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 13:54:46,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 13:54:46,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 13:54:46,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 13:54:46,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 13:54:46,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 13:54:46,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 13:54:46,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 13:54:46,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 13:54:46,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 13:54:46,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 13:54:46,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 13:54:46,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 13:54:46,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 13:54:46,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 13:54:46,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 13:54:46,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 13:54:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 13:54:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 13:54:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 13:54:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 13:54:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 13:54:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 13:54:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 13:54:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 13:54:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 13:54:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 13:54:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 13:54:46,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 13:54:46,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 13:54:46,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-27 13:54:46,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-27 13:54:46,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-27 13:54:46,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-27 13:54:46,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-27 13:54:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-27 13:54:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-27 13:54:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-27 13:54:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-27 13:54:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-27 13:54:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-27 13:54:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-27 13:54:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-27 13:54:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-27 13:54:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-27 13:54:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-27 13:54:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-27 13:54:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-27 13:54:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-27 13:54:46,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-27 13:54:46,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-27 13:54:46,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-27 13:54:46,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-27 13:54:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-27 13:54:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-27 13:54:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-27 13:54:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-27 13:54:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-27 13:54:46,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-27 13:54:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-27 13:54:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-27 13:54:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-27 13:54:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-27 13:54:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-27 13:54:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-27 13:54:46,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-27 13:54:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-27 13:54:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-27 13:54:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-27 13:54:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-27 13:54:46,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-27 13:54:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-27 13:54:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-27 13:54:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-27 13:54:46,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-27 13:54:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-27 13:54:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-27 13:54:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-27 13:54:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-27 13:54:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-27 13:54:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-27 13:54:46,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-27 13:54:46,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-27 13:54:46,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-27 13:54:46,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-27 13:54:46,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-27 13:54:46,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-27 13:54:46,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-27 13:54:46,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-27 13:54:46,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-27 13:54:46,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-27 13:54:46,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-27 13:54:46,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-27 13:54:46,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-27 13:54:46,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-27 13:54:46,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-27 13:54:46,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-27 13:54:46,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-27 13:54:46,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-27 13:54:46,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-27 13:54:46,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-27 13:54:46,738 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 13:54:46,738 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-27 13:54:46,738 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-27 13:54:47,318 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 758 ms to try 5000 different iterations.  Found a solution that moves 4681 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1841215456501712. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04174847935099808, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05971875724033418, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5201111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04174847935099808, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17941666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24541666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 13:54:47,344 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1043919302:81 , srv1156221557:81 , srv1681018455:81 , srv316896813:81 , srv785490953:81 , srv1633228380:82 , srv1069272992:83 , srv1173680790:83 , srv123296776:83 , srv1519941666:83 , srv1654262190:83 , srv1818249470:83 , srv2084897494:83 , srv211562863:83 , srv386148969:83 , srv484120592:83 , srv855541105:83 , srv1284514682:84 , srv136130081:84 , srv1905380746:84 , srv2086110549:84 , srv2118559680:84 , srv259713415:84 , srv393151695:84 , srv621850103:84 , srv74091037:84 , srv1185490534:85 , srv1187158199:85 , srv1329886908:85 , srv164020636:85 , srv173683906:85 , srv1801066186:85 , srv1817133068:85 , srv1917311342:85 , srv2134282624:85 , srv536238880:85 , srv553289064:85 , srv697155186:85 , srv711119170:85 , srv825943573:85 , srv111102991:86 , srv1119284339:86 , srv1232312247:86 , srv1423697713:86 , srv1634009801:86 , srv1779218622:86 , srv223295684:86 , srv31654380:86 , srv33442412:86 , srv399853691:86 , srv451288569:86 , srv56615975:86 , srv638864615:86 , srv841133863:86 , srv1049980706:87 , srv131977905:87 , srv1330482500:87 , srv1702124911:87 , srv1727710920:87 , srv1759292977:87 , srv1894769102:87 , srv1948567610:87 , srv2079630175:87 , srv235902957:87 , srv355245942:87 , srv466382273:87 , srv770623518:87 , srv829130356:87 , srv121250919:88 , srv1340684737:88 , srv1554293361:88 , srv1662367272:88 , srv1972251588:88 , srv1990676088:88 , srv2147236586:88 , srv302430793:88 , srv395768122:88 , srv431093254:88 , srv976521083:88 , srv1123393060:89 , srv1615845115:89 , srv1771182623:89 , srv1955910012:89 , srv373911620:89 , srv572238852:89 , srv675822337:89 , srv749418874:89 , srv798797433:89 , srv819507800:89 , srv1230855695:90 , srv1727162507:90 , srv1964488747:90 , srv9809780:90 , srv1716775237:91 , srv714503247:91 , srv1440285310:92 , srv1573441745:92 , srv943913165:92 , srv851447204:95 , srv2073649795:461 }
2023-03-27 13:54:47,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 13:54:47,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv749418874=87, srv819507800=91, srv1948567610=46, srv302430793=62, srv2134282624=57, srv2118559680=56, srv1615845115=25, srv798797433=90, srv829130356=93, srv1554293361=23, srv393151695=69, srv638864615=81, srv855541105=96, srv2073649795=51, srv74091037=86, srv825943573=92, srv572238852=79, srv841133863=94, srv1702124911=32, srv1654262190=29, srv1801066186=40, srv121250919=10, srv2147236586=58, srv1955910012=47, srv123296776=13, srv1964488747=48, srv553289064=77, srv770623518=88, srv1069272992=2, srv386148969=68, srv1173680790=7, srv395768122=70, srv1727162507=34, srv697155186=83, srv211562863=55, srv1573441745=24, srv1681018455=31, srv373911620=67, srv111102991=3, srv1329886908=16, srv399853691=71, srv943913165=97, srv466382273=74, srv1817133068=41, srv484120592=75, srv1634009801=27, srv714503247=85, srv1662367272=30, srv1771182623=38, srv223295684=59, srv851447204=95, srv1230855695=11, srv1119284339=4, srv1894769102=43, srv173683906=36, srv1917311342=45, srv2079630175=52, srv56615975=78, srv1905380746=44, srv355245942=66, srv1818249470=42, srv131977905=15, srv711119170=84, srv2086110549=54, srv1716775237=33, srv536238880=76, srv1123393060=5, srv1049980706=1, srv1423697713=20, srv431093254=72, srv1043919302=0, srv1972251588=49, srv1759292977=37, srv675822337=82, srv259713415=61, srv2084897494=53, srv976521083=98, srv1185490534=8, srv621850103=80, srv164020636=28, srv31654380=63, srv1779218622=39, srv1156221557=6, srv785490953=89, srv235902957=60, srv136130081=19, srv33442412=65, srv1284514682=14, srv9809780=99, srv1440285310=21, srv1633228380=26, srv451288569=73, srv1232312247=12, srv1340684737=18, srv1990676088=50, srv316896813=64, srv1519941666=22, srv1187158199=9, srv1727710920=35, srv1330482500=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 13:54:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 13:54:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 13:54:47,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 13:54:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 13:54:47,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 13:54:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 13:54:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 13:54:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 13:54:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 13:54:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 13:54:47,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 13:54:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 13:54:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 13:54:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 13:54:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 13:54:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 13:54:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 13:54:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 13:54:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 13:54:47,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 13:54:47,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 13:54:47,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 13:54:47,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 13:54:47,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 13:54:47,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 13:54:47,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 13:54:47,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 13:54:47,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 13:54:47,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 13:54:47,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 13:54:47,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-27 13:54:47,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-27 13:54:47,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-27 13:54:47,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-27 13:54:47,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-27 13:54:47,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-27 13:54:47,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-27 13:54:47,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-27 13:54:47,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-27 13:54:47,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-27 13:54:47,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-27 13:54:47,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-27 13:54:47,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-27 13:54:47,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-27 13:54:47,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-27 13:54:47,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-27 13:54:47,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-27 13:54:47,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-27 13:54:47,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-27 13:54:47,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-27 13:54:47,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-27 13:54:47,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-27 13:54:47,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-27 13:54:47,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-27 13:54:47,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-27 13:54:47,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-27 13:54:47,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-27 13:54:47,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-27 13:54:47,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-27 13:54:47,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-27 13:54:47,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-27 13:54:47,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-27 13:54:47,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-27 13:54:47,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-27 13:54:47,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-27 13:54:47,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-27 13:54:47,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-27 13:54:47,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-27 13:54:47,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-27 13:54:47,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-27 13:54:47,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-27 13:54:47,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-27 13:54:47,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-27 13:54:47,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-27 13:54:47,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-27 13:54:47,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-27 13:54:47,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-27 13:54:47,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-27 13:54:47,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-27 13:54:47,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-27 13:54:47,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-27 13:54:47,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-27 13:54:47,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-27 13:54:47,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-27 13:54:47,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-27 13:54:47,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-27 13:54:47,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-27 13:54:47,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-27 13:54:47,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-27 13:54:47,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-27 13:54:47,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-27 13:54:47,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-27 13:54:47,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-27 13:54:47,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-27 13:54:47,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-27 13:54:47,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-27 13:54:47,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-27 13:54:47,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-27 13:54:47,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-27 13:54:47,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-27 13:54:47,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-27 13:54:47,586 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04174847935099808, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05971875724033418, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04174847935099808, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17941666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24541666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 13:54:47,586 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-27 13:54:47,587 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18408876858761297, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04174847935099808, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05971875724033418, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04174847935099808, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17941666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24541666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-27 13:54:47,985 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 546 ms to try 5000 different iterations.  Found a solution that moves 2654 regions; Going from a computed imbalance of 0.18408876858761297 to a new imbalance of 0.016039450455671896. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024323237916260527); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03640710084782443, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29488888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024323237916260527); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 13:54:47,999 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1573441745:77 , srv851447204:77 , srv1440285310:80 , srv1615845115:80 , srv1716775237:80 , srv572238852:80 , srv714503247:80 , srv1123393060:81 , srv121250919:81 , srv1771182623:81 , srv943913165:81 , srv1230855695:82 , srv1330482500:82 , srv1955910012:82 , srv1964488747:82 , srv1972251588:82 , srv1990676088:82 , srv431093254:82 , srv56615975:82 , srv675822337:82 , srv1662367272:83 , srv1727162507:83 , srv1759292977:83 , srv395768122:83 , srv9809780:83 , srv1119284339:84 , srv131977905:84 , srv1340684737:84 , srv1894769102:84 , srv1948567610:84 , srv2147236586:84 , srv302430793:84 , srv355245942:84 , srv373911620:84 , srv749418874:84 , srv1049980706:85 , srv1232312247:85 , srv1554293361:85 , srv1702124911:85 , srv2079630175:85 , srv2086110549:85 , srv33442412:85 , srv451288569:85 , srv466382273:85 , srv770623518:85 , srv798797433:85 , srv819507800:85 , srv841133863:85 , srv976521083:85 , srv123296776:86 , srv1633228380:86 , srv1727710920:86 , srv173683906:86 , srv1779218622:86 , srv223295684:86 , srv399853691:86 , srv536238880:86 , srv553289064:86 , srv638864615:86 , srv711119170:86 , srv74091037:86 , srv829130356:86 , srv1173680790:87 , srv1187158199:87 , srv136130081:87 , srv1423697713:87 , srv1905380746:87 , srv1917311342:87 , srv259713415:87 , srv111102991:88 , srv1284514682:88 , srv1654262190:88 , srv1817133068:88 , srv2134282624:88 , srv235902957:88 , srv386148969:88 , srv484120592:88 , srv621850103:88 , srv825943573:88 , srv1634009801:89 , srv1681018455:89 , srv1818249470:89 , srv211562863:89 , srv2118559680:89 , srv31654380:89 , srv1156221557:90 , srv1185490534:90 , srv1329886908:90 , srv2084897494:90 , srv316896813:90 , srv393151695:90 , srv785490953:90 , srv855541105:90 , srv1043919302:91 , srv1519941666:91 , srv1801066186:91 , srv1069272992:92 , srv164020636:92 , srv697155186:92 , srv2073649795:529 }
2023-03-27 13:54:48,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 13:54:48,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv749418874=87, srv819507800=91, srv1948567610=46, srv302430793=62, srv2134282624=57, srv2118559680=56, srv1615845115=25, srv798797433=90, srv829130356=93, srv1554293361=23, srv393151695=69, srv638864615=81, srv855541105=96, srv2073649795=51, srv74091037=86, srv825943573=92, srv572238852=79, srv841133863=94, srv1702124911=32, srv1654262190=29, srv1801066186=40, srv121250919=10, srv2147236586=58, srv1955910012=47, srv123296776=13, srv1964488747=48, srv553289064=77, srv770623518=88, srv1069272992=2, srv386148969=68, srv1173680790=7, srv395768122=70, srv1727162507=34, srv697155186=83, srv211562863=55, srv1573441745=24, srv1681018455=31, srv373911620=67, srv111102991=3, srv1329886908=16, srv399853691=71, srv943913165=97, srv466382273=74, srv1817133068=41, srv484120592=75, srv1634009801=27, srv714503247=85, srv1662367272=30, srv1771182623=38, srv223295684=59, srv851447204=95, srv1230855695=11, srv1119284339=4, srv1894769102=43, srv173683906=36, srv1917311342=45, srv2079630175=52, srv56615975=78, srv1905380746=44, srv355245942=66, srv1818249470=42, srv131977905=15, srv711119170=84, srv2086110549=54, srv1716775237=33, srv536238880=76, srv1123393060=5, srv1049980706=1, srv1423697713=20, srv431093254=72, srv1043919302=0, srv1972251588=49, srv1759292977=37, srv675822337=82, srv259713415=61, srv2084897494=53, srv976521083=98, srv1185490534=8, srv621850103=80, srv164020636=28, srv31654380=63, srv1779218622=39, srv1156221557=6, srv785490953=89, srv235902957=60, srv136130081=19, srv33442412=65, srv1284514682=14, srv9809780=99, srv1440285310=21, srv1633228380=26, srv451288569=73, srv1232312247=12, srv1340684737=18, srv1990676088=50, srv316896813=64, srv1519941666=22, srv1187158199=9, srv1727710920=35, srv1330482500=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 13:54:48,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 13:54:48,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 13:54:48,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 13:54:48,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 13:54:48,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 13:54:48,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 13:54:48,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 13:54:48,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 13:54:48,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 13:54:48,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 13:54:48,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 13:54:48,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 13:54:48,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 13:54:48,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 13:54:48,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 13:54:48,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 13:54:48,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 13:54:48,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 13:54:48,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 13:54:48,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 13:54:48,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 13:54:48,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 13:54:48,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 13:54:48,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 13:54:48,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 13:54:48,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 13:54:48,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 13:54:48,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 13:54:48,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 13:54:48,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 13:54:48,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-27 13:54:48,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-27 13:54:48,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-27 13:54:48,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-27 13:54:48,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-27 13:54:48,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-27 13:54:48,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-27 13:54:48,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-27 13:54:48,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-27 13:54:48,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-27 13:54:48,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-27 13:54:48,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-27 13:54:48,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-27 13:54:48,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-27 13:54:48,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-27 13:54:48,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-27 13:54:48,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-27 13:54:48,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-27 13:54:48,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-27 13:54:48,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-27 13:54:48,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-27 13:54:48,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-27 13:54:48,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-27 13:54:48,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-27 13:54:48,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-27 13:54:48,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-27 13:54:48,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-27 13:54:48,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-27 13:54:48,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-27 13:54:48,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-27 13:54:48,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-27 13:54:48,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-27 13:54:48,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-27 13:54:48,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-27 13:54:48,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-27 13:54:48,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-27 13:54:48,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-27 13:54:48,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-27 13:54:48,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-27 13:54:48,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-27 13:54:48,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-27 13:54:48,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-27 13:54:48,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-27 13:54:48,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-27 13:54:48,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-27 13:54:48,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-27 13:54:48,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-27 13:54:48,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-27 13:54:48,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-27 13:54:48,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-27 13:54:48,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-27 13:54:48,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-27 13:54:48,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-27 13:54:48,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-27 13:54:48,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-27 13:54:48,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-27 13:54:48,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-27 13:54:48,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-27 13:54:48,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-27 13:54:48,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-27 13:54:48,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-27 13:54:48,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-27 13:54:48,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-27 13:54:48,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-27 13:54:48,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-27 13:54:48,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-27 13:54:48,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-27 13:54:48,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-27 13:54:48,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-27 13:54:48,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-27 13:54:48,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-27 13:54:48,173 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024323237916260527); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03640710084782443, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024323237916260527); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 13:54:48,173 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-27 13:54:48,173 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016020866750474403, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024323237916260527); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03640710084782443, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024323237916260527); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-27 13:54:48,490 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 427 ms to try 5000 different iterations.  Found a solution that moves 972 regions; Going from a computed imbalance of 0.016020866750474403 to a new imbalance of 0.0010362550801467126. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0066647958467430145); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006564879518897745); MoveCostFunction : (multiplier=7.0, imbalance=0.108, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0066647958467430145); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 13:54:48,494 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv164020636:80 , srv697155186:80 , srv1801066186:81 , srv235902957:81 , srv1185490534:82 , srv31654380:82 , srv1069272992:83 , srv1329886908:83 , srv1634009801:83 , srv855541105:83 , srv111102991:84 , srv1423697713:84 , srv1519941666:84 , srv1817133068:84 , srv2084897494:84 , srv2118559680:84 , srv373911620:84 , srv393151695:84 , srv798797433:84 , srv819507800:84 , srv825943573:84 , srv943913165:84 , srv9809780:84 , srv1043919302:85 , srv1187158199:85 , srv1230855695:85 , srv1284514682:85 , srv1340684737:85 , srv1554293361:85 , srv1702124911:85 , srv1727162507:85 , srv1727710920:85 , srv1818249470:85 , srv1955910012:85 , srv1964488747:85 , srv2134282624:85 , srv223295684:85 , srv466382273:85 , srv621850103:85 , srv770623518:85 , srv829130356:85 , srv851447204:85 , srv1049980706:86 , srv1173680790:86 , srv1232312247:86 , srv131977905:86 , srv136130081:86 , srv1440285310:86 , srv1662367272:86 , srv173683906:86 , srv1779218622:86 , srv1905380746:86 , srv1917311342:86 , srv2079630175:86 , srv211562863:86 , srv2147236586:86 , srv302430793:86 , srv316896813:86 , srv33442412:86 , srv355245942:86 , srv386148969:86 , srv399853691:86 , srv451288569:86 , srv484120592:86 , srv536238880:86 , srv553289064:86 , srv638864615:86 , srv714503247:86 , srv749418874:86 , srv785490953:86 , srv841133863:86 , srv1119284339:87 , srv1123393060:87 , srv1156221557:87 , srv1654262190:87 , srv1681018455:87 , srv1716775237:87 , srv1771182623:87 , srv1894769102:87 , srv1948567610:87 , srv1990676088:87 , srv259713415:87 , srv395768122:87 , srv431093254:87 , srv711119170:87 , srv74091037:87 , srv976521083:87 , srv123296776:88 , srv1615845115:88 , srv1759292977:88 , srv1972251588:88 , srv2086110549:88 , srv572238852:88 , srv121250919:89 , srv1330482500:89 , srv1573441745:89 , srv1633228380:89 , srv56615975:89 , srv675822337:89 , srv2073649795:527 }
2023-03-27 13:54:48,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 13:54:48,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv749418874=87, srv819507800=91, srv1948567610=46, srv302430793=62, srv2134282624=57, srv2118559680=56, srv1615845115=25, srv798797433=90, srv829130356=93, srv1554293361=23, srv393151695=69, srv638864615=81, srv855541105=96, srv2073649795=51, srv74091037=86, srv825943573=92, srv572238852=79, srv841133863=94, srv1702124911=32, srv1654262190=29, srv1801066186=40, srv121250919=10, srv2147236586=58, srv1955910012=47, srv123296776=13, srv1964488747=48, srv553289064=77, srv770623518=88, srv1069272992=2, srv386148969=68, srv1173680790=7, srv395768122=70, srv1727162507=34, srv697155186=83, srv211562863=55, srv1573441745=24, srv1681018455=31, srv373911620=67, srv111102991=3, srv1329886908=16, srv399853691=71, srv943913165=97, srv466382273=74, srv1817133068=41, srv484120592=75, srv1634009801=27, srv714503247=85, srv1662367272=30, srv1771182623=38, srv223295684=59, srv851447204=95, srv1230855695=11, srv1119284339=4, srv1894769102=43, srv173683906=36, srv1917311342=45, srv2079630175=52, srv56615975=78, srv1905380746=44, srv355245942=66, srv1818249470=42, srv131977905=15, srv711119170=84, srv2086110549=54, srv1716775237=33, srv536238880=76, srv1123393060=5, srv1049980706=1, srv1423697713=20, srv431093254=72, srv1043919302=0, srv1972251588=49, srv1759292977=37, srv675822337=82, srv259713415=61, srv2084897494=53, srv976521083=98, srv1185490534=8, srv621850103=80, srv164020636=28, srv31654380=63, srv1779218622=39, srv1156221557=6, srv785490953=89, srv235902957=60, srv136130081=19, srv33442412=65, srv1284514682=14, srv9809780=99, srv1440285310=21, srv1633228380=26, srv451288569=73, srv1232312247=12, srv1340684737=18, srv1990676088=50, srv316896813=64, srv1519941666=22, srv1187158199=9, srv1727710920=35, srv1330482500=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 13:54:48,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 13:54:48,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 13:54:48,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 13:54:48,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 13:54:48,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 13:54:48,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 13:54:48,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 13:54:48,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 13:54:48,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 13:54:48,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 13:54:48,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 13:54:48,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 13:54:48,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 13:54:48,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 13:54:48,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 13:54:48,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 13:54:48,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 13:54:48,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 13:54:48,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 13:54:48,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 13:54:48,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 13:54:48,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 13:54:48,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 13:54:48,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 13:54:48,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 13:54:48,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 13:54:48,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 13:54:48,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 13:54:48,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 13:54:48,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 13:54:48,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-27 13:54:48,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-27 13:54:48,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-27 13:54:48,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-27 13:54:48,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-27 13:54:48,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-27 13:54:48,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-27 13:54:48,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-27 13:54:48,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-27 13:54:48,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-27 13:54:48,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-27 13:54:48,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-27 13:54:48,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-27 13:54:48,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-27 13:54:48,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-27 13:54:48,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-27 13:54:48,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-27 13:54:48,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-27 13:54:48,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-27 13:54:48,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-27 13:54:48,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-27 13:54:48,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-27 13:54:48,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-27 13:54:48,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-27 13:54:48,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-27 13:54:48,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-27 13:54:48,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-27 13:54:48,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-27 13:54:48,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-27 13:54:48,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-27 13:54:48,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-27 13:54:48,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-27 13:54:48,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-27 13:54:48,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-27 13:54:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-27 13:54:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-27 13:54:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-27 13:54:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-27 13:54:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-27 13:54:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-27 13:54:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-27 13:54:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-27 13:54:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-27 13:54:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-27 13:54:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-27 13:54:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-27 13:54:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-27 13:54:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-27 13:54:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-27 13:54:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-27 13:54:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-27 13:54:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-27 13:54:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-27 13:54:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-27 13:54:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-27 13:54:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-27 13:54:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-27 13:54:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-27 13:54:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-27 13:54:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-27 13:54:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-27 13:54:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-27 13:54:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-27 13:54:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-27 13:54:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-27 13:54:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-27 13:54:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-27 13:54:48,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-27 13:54:48,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-27 13:54:48,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-27 13:54:48,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-27 13:54:48,703 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0066647958467430145); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006564879518897745); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0066647958467430145); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 13:54:48,704 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-27 13:54:48,704 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0010294489906772454, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0066647958467430145); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006564879518897745); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0066647958467430145); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-27 13:54:49,210 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 639 ms to try 5000 different iterations.  Found a solution that moves 258 regions; Going from a computed imbalance of 0.0010294489906772454 to a new imbalance of 1.2965815196837248E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.028666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 13:54:49,211 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1440285310:83 , srv1727710920:83 , srv1779218622:83 , srv675822337:83 , srv749418874:83 , srv976521083:83 , srv1049980706:84 , srv1069272992:84 , srv111102991:84 , srv1156221557:84 , srv121250919:84 , srv1232312247:84 , srv1329886908:84 , srv1330482500:84 , srv1554293361:84 , srv1573441745:84 , srv1634009801:84 , srv1654262190:84 , srv1681018455:84 , srv1716775237:84 , srv1727162507:84 , srv1759292977:84 , srv1894769102:84 , srv1905380746:84 , srv1917311342:84 , srv1948567610:84 , srv1972251588:84 , srv2079630175:84 , srv211562863:84 , srv2134282624:84 , srv2147236586:84 , srv259713415:84 , srv302430793:84 , srv393151695:84 , srv395768122:84 , srv399853691:84 , srv638864615:84 , srv711119170:84 , srv798797433:84 , srv819507800:84 , srv829130356:84 , srv1043919302:85 , srv1119284339:85 , srv1123393060:85 , srv1185490534:85 , srv1187158199:85 , srv1230855695:85 , srv123296776:85 , srv1284514682:85 , srv131977905:85 , srv1340684737:85 , srv136130081:85 , srv1423697713:85 , srv1519941666:85 , srv1615845115:85 , srv1633228380:85 , srv1662367272:85 , srv1702124911:85 , srv173683906:85 , srv1771182623:85 , srv1801066186:85 , srv1817133068:85 , srv1818249470:85 , srv1964488747:85 , srv1990676088:85 , srv2084897494:85 , srv2086110549:85 , srv2118559680:85 , srv223295684:85 , srv31654380:85 , srv316896813:85 , srv33442412:85 , srv355245942:85 , srv373911620:85 , srv386148969:85 , srv431093254:85 , srv451288569:85 , srv484120592:85 , srv536238880:85 , srv553289064:85 , srv56615975:85 , srv572238852:85 , srv621850103:85 , srv697155186:85 , srv714503247:85 , srv74091037:85 , srv770623518:85 , srv785490953:85 , srv825943573:85 , srv851447204:85 , srv943913165:85 , srv9809780:85 , srv1173680790:86 , srv164020636:86 , srv1955910012:86 , srv235902957:86 , srv466382273:86 , srv841133863:86 , srv855541105:86 , srv2073649795:625 }
2023-03-27 13:54:49,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 13:54:49,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv749418874=87, srv819507800=91, srv1948567610=46, srv302430793=62, srv2134282624=57, srv2118559680=56, srv1615845115=25, srv798797433=90, srv829130356=93, srv1554293361=23, srv393151695=69, srv638864615=81, srv855541105=96, srv2073649795=51, srv74091037=86, srv825943573=92, srv572238852=79, srv841133863=94, srv1702124911=32, srv1654262190=29, srv1801066186=40, srv121250919=10, srv2147236586=58, srv1955910012=47, srv123296776=13, srv1964488747=48, srv553289064=77, srv770623518=88, srv1069272992=2, srv386148969=68, srv1173680790=7, srv395768122=70, srv1727162507=34, srv697155186=83, srv211562863=55, srv1573441745=24, srv1681018455=31, srv373911620=67, srv111102991=3, srv1329886908=16, srv399853691=71, srv943913165=97, srv466382273=74, srv1817133068=41, srv484120592=75, srv1634009801=27, srv714503247=85, srv1662367272=30, srv1771182623=38, srv223295684=59, srv851447204=95, srv1230855695=11, srv1119284339=4, srv1894769102=43, srv173683906=36, srv1917311342=45, srv2079630175=52, srv56615975=78, srv1905380746=44, srv355245942=66, srv1818249470=42, srv131977905=15, srv711119170=84, srv2086110549=54, srv1716775237=33, srv536238880=76, srv1123393060=5, srv1049980706=1, srv1423697713=20, srv431093254=72, srv1043919302=0, srv1972251588=49, srv1759292977=37, srv675822337=82, srv259713415=61, srv2084897494=53, srv976521083=98, srv1185490534=8, srv621850103=80, srv164020636=28, srv31654380=63, srv1779218622=39, srv1156221557=6, srv785490953=89, srv235902957=60, srv136130081=19, srv33442412=65, srv1284514682=14, srv9809780=99, srv1440285310=21, srv1633228380=26, srv451288569=73, srv1232312247=12, srv1340684737=18, srv1990676088=50, srv316896813=64, srv1519941666=22, srv1187158199=9, srv1727710920=35, srv1330482500=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 13:54:49,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 13:54:49,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 13:54:49,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 13:54:49,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 13:54:49,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 13:54:49,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 13:54:49,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 13:54:49,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 13:54:49,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 13:54:49,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 13:54:49,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 13:54:49,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 13:54:49,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 13:54:49,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 13:54:49,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 13:54:49,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 13:54:49,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 13:54:49,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 13:54:49,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 13:54:49,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 13:54:49,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 13:54:49,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 13:54:49,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 13:54:49,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 13:54:49,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 13:54:49,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 13:54:49,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 13:54:49,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 13:54:49,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 13:54:49,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 13:54:49,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-27 13:54:49,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-27 13:54:49,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-27 13:54:49,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-27 13:54:49,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-27 13:54:49,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-27 13:54:49,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-27 13:54:49,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-27 13:54:49,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-27 13:54:49,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-27 13:54:49,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-27 13:54:49,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-27 13:54:49,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-27 13:54:49,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-27 13:54:49,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-27 13:54:49,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-27 13:54:49,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-27 13:54:49,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-27 13:54:49,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-27 13:54:49,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-27 13:54:49,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-27 13:54:49,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-27 13:54:49,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-27 13:54:49,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-27 13:54:49,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-27 13:54:49,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-27 13:54:49,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-27 13:54:49,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-27 13:54:49,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-27 13:54:49,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-27 13:54:49,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-27 13:54:49,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-27 13:54:49,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-27 13:54:49,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-27 13:54:49,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-27 13:54:49,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-27 13:54:49,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-27 13:54:49,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-27 13:54:49,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-27 13:54:49,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-27 13:54:49,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-27 13:54:49,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-27 13:54:49,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-27 13:54:49,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-27 13:54:49,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-27 13:54:49,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-27 13:54:49,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-27 13:54:49,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-27 13:54:49,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-27 13:54:49,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-27 13:54:49,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-27 13:54:49,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-27 13:54:49,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-27 13:54:49,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-27 13:54:49,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-27 13:54:49,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-27 13:54:49,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-27 13:54:49,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-27 13:54:49,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-27 13:54:49,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-27 13:54:49,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-27 13:54:49,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-27 13:54:49,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-27 13:54:49,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-27 13:54:49,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-27 13:54:49,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-27 13:54:49,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-27 13:54:49,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-27 13:54:49,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-27 13:54:49,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-27 13:54:49,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-27 13:54:49,388 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.1159260584571282E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 13:54:49,388 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1440285310:83 , srv1727710920:83 , srv1779218622:83 , srv675822337:83 , srv749418874:83 , srv976521083:83 , srv1049980706:84 , srv1069272992:84 , srv111102991:84 , srv1156221557:84 , srv121250919:84 , srv1232312247:84 , srv1329886908:84 , srv1330482500:84 , srv1554293361:84 , srv1573441745:84 , srv1634009801:84 , srv1654262190:84 , srv1681018455:84 , srv1716775237:84 , srv1727162507:84 , srv1759292977:84 , srv1894769102:84 , srv1905380746:84 , srv1917311342:84 , srv1948567610:84 , srv1972251588:84 , srv2079630175:84 , srv211562863:84 , srv2134282624:84 , srv2147236586:84 , srv259713415:84 , srv302430793:84 , srv393151695:84 , srv395768122:84 , srv399853691:84 , srv638864615:84 , srv711119170:84 , srv798797433:84 , srv819507800:84 , srv829130356:84 , srv1043919302:85 , srv1119284339:85 , srv1123393060:85 , srv1185490534:85 , srv1187158199:85 , srv1230855695:85 , srv123296776:85 , srv1284514682:85 , srv131977905:85 , srv1340684737:85 , srv136130081:85 , srv1423697713:85 , srv1519941666:85 , srv1615845115:85 , srv1633228380:85 , srv1662367272:85 , srv1702124911:85 , srv173683906:85 , srv1771182623:85 , srv1801066186:85 , srv1817133068:85 , srv1818249470:85 , srv1964488747:85 , srv1990676088:85 , srv2084897494:85 , srv2086110549:85 , srv2118559680:85 , srv223295684:85 , srv31654380:85 , srv316896813:85 , srv33442412:85 , srv355245942:85 , srv373911620:85 , srv386148969:85 , srv431093254:85 , srv451288569:85 , srv484120592:85 , srv536238880:85 , srv553289064:85 , srv56615975:85 , srv572238852:85 , srv621850103:85 , srv697155186:85 , srv714503247:85 , srv74091037:85 , srv770623518:85 , srv785490953:85 , srv825943573:85 , srv851447204:85 , srv943913165:85 , srv9809780:85 , srv1173680790:86 , srv164020636:86 , srv1955910012:86 , srv235902957:86 , srv466382273:86 , srv841133863:86 , srv855541105:86 , srv2073649795:625 }
2023-03-27 13:54:49,431 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=1320 (was 1320), ProcessCount=205 (was 207), AvailableMemoryMB=8922 (was 9617)
2023-03-27 13:54:49,454 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=1320, ProcessCount=205, AvailableMemoryMB=8918
2023-03-27 13:54:49,455 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-27 13:54:49,455 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-27 13:54:49,471 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1036383422:84 , srv1106442592:84 , srv12912122:84 , srv1381570267:84 , srv1419427241:84 , srv1602677884:84 , srv1614431238:84 , srv1674024582:84 , srv1674519489:84 , srv1776646334:84 , srv1795093447:84 , srv1803022714:84 , srv1994065241:84 , srv2012719385:84 , srv2036174092:84 , srv2110836535:84 , srv2121730630:84 , srv214631533:84 , srv224124543:84 , srv307500287:84 , srv313381716:84 , srv403529725:84 , srv415072600:84 , srv441880105:84 , srv531910872:84 , srv559335129:84 , srv612650393:84 , srv614147037:84 , srv945648193:84 , srv1645708361:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-27 13:54:49,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 13:54:49,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2012719385=14, srv614147037=28, srv1381570267=3, srv307500287=20, srv945648193=29, srv1645708361=7, srv441880105=24, srv1419427241=4, srv531910872=25, srv1994065241=13, srv1036383422=0, srv403529725=22, srv1614431238=6, srv313381716=21, srv1776646334=10, srv1602677884=5, srv2036174092=15, srv2121730630=17, srv224124543=19, srv1674024582=8, srv1674519489=9, srv1795093447=11, srv12912122=2, srv1803022714=12, srv1106442592=1, srv214631533=18, srv612650393=27, srv415072600=23, srv2110836535=16, srv559335129=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 13:54:49,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 13:54:49,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 13:54:49,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 13:54:49,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 13:54:49,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 13:54:49,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 13:54:49,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 13:54:49,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 13:54:49,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 13:54:49,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 13:54:49,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 13:54:49,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 13:54:49,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 13:54:49,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 13:54:49,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 13:54:49,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 13:54:49,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 13:54:49,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 13:54:49,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 13:54:49,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 13:54:49,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 13:54:49,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 13:54:49,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 13:54:49,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 13:54:49,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 13:54:49,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 13:54:49,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 13:54:49,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 13:54:49,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 13:54:49,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 13:54:49,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-27 13:54:49,526 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0999121491907096, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 13:54:49,526 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912507863183948, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0999121491907096, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-27 13:56:49,501 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 7322879 different iterations.  Found a solution that moves 1839 regions; Going from a computed imbalance of 0.9912507863183948 to a new imbalance of 4.292317741546655E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6811111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 13:56:49,506 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1036383422:90 , srv1106442592:90 , srv12912122:90 , srv1381570267:90 , srv1419427241:90 , srv1602677884:90 , srv1614431238:90 , srv1645708361:90 , srv1674024582:90 , srv1674519489:90 , srv1776646334:90 , srv1795093447:90 , srv1803022714:90 , srv1994065241:90 , srv2012719385:90 , srv2036174092:90 , srv2110836535:90 , srv2121730630:90 , srv214631533:90 , srv224124543:90 , srv307500287:90 , srv313381716:90 , srv403529725:90 , srv415072600:90 , srv441880105:90 , srv531910872:90 , srv559335129:90 , srv612650393:90 , srv614147037:90 , srv945648193:90 }
2023-03-27 13:56:49,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-27 13:56:49,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2012719385=14, srv614147037=28, srv1381570267=3, srv307500287=20, srv945648193=29, srv1645708361=7, srv441880105=24, srv1419427241=4, srv531910872=25, srv1994065241=13, srv1036383422=0, srv403529725=22, srv1614431238=6, srv313381716=21, srv1776646334=10, srv1602677884=5, srv2036174092=15, srv2121730630=17, srv224124543=19, srv1674024582=8, srv1674519489=9, srv1795093447=11, srv12912122=2, srv1803022714=12, srv1106442592=1, srv214631533=18, srv612650393=27, srv415072600=23, srv2110836535=16, srv559335129=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-27 13:56:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-27 13:56:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-27 13:56:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-27 13:56:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-27 13:56:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-27 13:56:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-27 13:56:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-27 13:56:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-27 13:56:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-27 13:56:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-27 13:56:49,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-27 13:56:49,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-27 13:56:49,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-27 13:56:49,523 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-27 13:56:49,523 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1036383422:90 , srv1106442592:90 , srv12912122:90 , srv1381570267:90 , srv1419427241:90 , srv1602677884:90 , srv1614431238:90 , srv1645708361:90 , srv1674024582:90 , srv1674519489:90 , srv1776646334:90 , srv1795093447:90 , srv1803022714:90 , srv1994065241:90 , srv2012719385:90 , srv2036174092:90 , srv2110836535:90 , srv2121730630:90 , srv214631533:90 , srv224124543:90 , srv307500287:90 , srv313381716:90 , srv403529725:90 , srv415072600:90 , srv441880105:90 , srv531910872:90 , srv559335129:90 , srv612650393:90 , srv614147037:90 , srv945648193:90 }
2023-03-27 13:56:49,538 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=714 (was 1320), ProcessCount=180 (was 205), AvailableMemoryMB=9229 (was 8918) - AvailableMemoryMB LEAK? -