2023-03-08 19:54:15,747 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-08 19:54:16,051 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-08 19:54:16,162 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-08 19:54:16,223 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-08 19:54:16,231 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-08 19:54:16,243 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=219, ProcessCount=170, AvailableMemoryMB=8980
2023-03-08 19:54:16,245 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-08 19:54:16,246 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-08 19:54:16,366 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1007581474:84 , srv101724606:84 , srv1024671389:84 , srv1048792229:84 , srv1058339753:84 , srv1078594524:84 , srv1093796022:84 , srv1137241657:84 , srv118934748:84 , srv1211679586:84 , srv1214471393:84 , srv1217808268:84 , srv1224119572:84 , srv1232139054:84 , srv1232472006:84 , srv1242590913:84 , srv1274829392:84 , srv1289022739:84 , srv1323890131:84 , srv1341245919:84 , srv1363386838:84 , srv1375568840:84 , srv1426451057:84 , srv1426556588:84 , srv1436563639:84 , srv1454086126:84 , srv1480181004:84 , srv1488949678:84 , srv1500347950:84 , srv1506040424:84 , srv1590130185:84 , srv1592365101:84 , srv1595071430:84 , srv1621683733:84 , srv1716094262:84 , srv1728200542:84 , srv1752399437:84 , srv1775275062:84 , srv1790529274:84 , srv1816435330:84 , srv1818987872:84 , srv1833282407:84 , srv1847645502:84 , srv188389411:84 , srv1913479682:84 , srv1937078817:84 , srv1937204240:84 , srv1970368725:84 , srv1977648513:84 , srv1986652841:84 , srv1996668886:84 , srv2008676383:84 , srv2017481343:84 , srv2031118968:84 , srv2033271374:84 , srv2045300072:84 , srv2055645899:84 , srv2085781652:84 , srv233799912:84 , srv234000162:84 , srv268119056:84 , srv284121269:84 , srv302970796:84 , srv323821709:84 , srv339875449:84 , srv374446587:84 , srv378682709:84 , srv394011465:84 , srv396762648:84 , srv407770218:84 , srv416494389:84 , srv423309836:84 , srv452331273:84 , srv478931098:84 , srv488530593:84 , srv553610373:84 , srv572474164:84 , srv587121822:84 , srv632295599:84 , srv656230049:84 , srv663175520:84 , srv663880676:84 , srv670847770:84 , srv734017783:84 , srv744728030:84 , srv744931667:84 , srv766743637:84 , srv769260112:84 , srv769528735:84 , srv788122441:84 , srv78818348:84 , srv809924504:84 , srv822795415:84 , srv833026450:84 , srv85673404:84 , srv865664472:84 , srv902598655:84 , srv913474323:84 , srv941741303:84 , srv2125508707:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-08 19:54:16,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-08 19:54:16,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1426451057=22, srv339875449=65, srv284121269=62, srv1341245919=19, srv2031118968=53, srv1007581474=0, srv1093796022=6, srv1289022739=17, srv656230049=80, srv188389411=43, srv2085781652=57, srv234000162=60, srv396762648=69, srv394011465=68, srv769528735=89, srv2055645899=56, srv323821709=64, srv663175520=81, srv101724606=1, srv1728200542=35, srv233799912=59, srv1078594524=5, srv1818987872=40, srv1048792229=3, srv423309836=72, srv744728030=85, srv378682709=67, srv374446587=66, srv416494389=71, srv1232472006=14, srv78818348=91, srv1363386838=20, srv788122441=90, srv913474323=98, srv2045300072=55, srv268119056=61, srv766743637=87, srv118934748=8, srv809924504=92, srv1913479682=44, srv2008676383=51, srv407770218=70, srv865664472=96, srv1058339753=4, srv1426556588=23, srv572474164=77, srv1506040424=29, srv1833282407=41, srv1937204240=46, srv1937078817=45, srv1214471393=10, srv1977648513=48, srv2125508707=58, srv1590130185=30, srv1274829392=16, srv1488949678=27, srv553610373=76, srv663880676=82, srv1816435330=39, srv1970368725=47, srv1211679586=9, srv302970796=63, srv632295599=79, srv734017783=84, srv478931098=74, srv1217808268=11, srv769260112=88, srv1752399437=36, srv2033271374=54, srv1137241657=7, srv1232139054=13, srv833026450=94, srv1024671389=2, srv1775275062=37, srv85673404=95, srv1790529274=38, srv1847645502=42, srv941741303=99, srv744931667=86, srv488530593=75, srv1500347950=28, srv1224119572=12, srv822795415=93, srv2017481343=52, srv1242590913=15, srv1323890131=18, srv1454086126=25, srv1480181004=26, srv1595071430=32, srv1621683733=33, srv1436563639=24, srv1996668886=50, srv670847770=83, srv587121822=78, srv1375568840=21, srv1716094262=34, srv1986652841=49, srv1592365101=31, srv902598655=97, srv452331273=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-08 19:54:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-08 19:54:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-08 19:54:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-08 19:54:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-08 19:54:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-08 19:54:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-08 19:54:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-08 19:54:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-08 19:54:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-08 19:54:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-08 19:54:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-08 19:54:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-08 19:54:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-08 19:54:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-08 19:54:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-08 19:54:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-08 19:54:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-08 19:54:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-08 19:54:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-08 19:54:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-08 19:54:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-08 19:54:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-08 19:54:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-08 19:54:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-08 19:54:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-08 19:54:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-08 19:54:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-08 19:54:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-08 19:54:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-08 19:54:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-08 19:54:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-08 19:54:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-08 19:54:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-08 19:54:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-08 19:54:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-08 19:54:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-08 19:54:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-08 19:54:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-08 19:54:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-08 19:54:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-08 19:54:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-08 19:54:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-08 19:54:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-08 19:54:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-08 19:54:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-08 19:54:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-08 19:54:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-08 19:54:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-08 19:54:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-08 19:54:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-08 19:54:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-08 19:54:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-08 19:54:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-08 19:54:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-08 19:54:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-08 19:54:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-08 19:54:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-08 19:54:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-08 19:54:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-08 19:54:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-08 19:54:16,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-08 19:54:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-08 19:54:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-08 19:54:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-08 19:54:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-08 19:54:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-08 19:54:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-08 19:54:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-08 19:54:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-08 19:54:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-08 19:54:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-08 19:54:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-08 19:54:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-08 19:54:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-08 19:54:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-08 19:54:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-08 19:54:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-08 19:54:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-08 19:54:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-08 19:54:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-08 19:54:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-08 19:54:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-08 19:54:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-08 19:54:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-08 19:54:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-08 19:54:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-08 19:54:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-08 19:54:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-08 19:54:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-08 19:54:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-08 19:54:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-08 19:54:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-08 19:54:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-08 19:54:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-08 19:54:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-08 19:54:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-08 19:54:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-08 19:54:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-08 19:54:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-08 19:54:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-08 19:54:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-08 19:54:16,684 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-08 19:54:16,685 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-08 19:54:16,685 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-08 19:54:17,073 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 566 ms to try 5000 different iterations.  Found a solution that moves 4744 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1744676454238159. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039036596193499765, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051203254666475594, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5271111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039036596193499765, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16941666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23875, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-08 19:54:17,093 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1078594524:83 , srv1341245919:83 , srv1426451057:83 , srv1986652841:83 , srv2031118968:83 , srv2055645899:83 , srv323821709:83 , srv396762648:83 , srv656230049:83 , srv744728030:83 , srv1224119572:84 , srv1716094262:84 , srv1728200542:84 , srv1818987872:84 , srv2033271374:84 , srv663175520:84 , srv734017783:84 , srv822795415:84 , srv85673404:84 , srv913474323:84 , srv941741303:84 , srv1214471393:85 , srv1242590913:85 , srv1274829392:85 , srv1375568840:85 , srv1590130185:85 , srv1621683733:85 , srv1970368725:85 , srv423309836:85 , srv587121822:85 , srv632295599:85 , srv788122441:85 , srv809924504:85 , srv1007581474:86 , srv1093796022:86 , srv1211679586:86 , srv1217808268:86 , srv1426556588:86 , srv1752399437:86 , srv1833282407:86 , srv188389411:86 , srv1913479682:86 , srv1937078817:86 , srv1977648513:86 , srv2085781652:86 , srv233799912:86 , srv284121269:86 , srv302970796:86 , srv339875449:86 , srv394011465:86 , srv452331273:86 , srv572474164:86 , srv101724606:87 , srv1058339753:87 , srv1137241657:87 , srv1232139054:87 , srv1289022739:87 , srv1363386838:87 , srv1436563639:87 , srv1454086126:87 , srv1488949678:87 , srv1506040424:87 , srv1592365101:87 , srv1790529274:87 , srv1847645502:87 , srv1996668886:87 , srv416494389:87 , srv478931098:87 , srv553610373:87 , srv769260112:87 , srv769528735:87 , srv78818348:87 , srv865664472:87 , srv902598655:87 , srv1323890131:88 , srv1775275062:88 , srv1937204240:88 , srv2008676383:88 , srv2045300072:88 , srv234000162:88 , srv488530593:88 , srv663880676:88 , srv670847770:88 , srv1024671389:89 , srv1480181004:89 , srv1500347950:89 , srv1595071430:89 , srv2017481343:89 , srv268119056:89 , srv407770218:89 , srv1232472006:90 , srv378682709:90 , srv1048792229:91 , srv118934748:91 , srv1816435330:91 , srv744931667:91 , srv766743637:93 , srv833026450:93 , srv374446587:96 , srv2125508707:437 }
2023-03-08 19:54:17,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-08 19:54:17,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1426451057=22, srv339875449=65, srv284121269=62, srv1341245919=19, srv2031118968=53, srv1007581474=0, srv1093796022=6, srv1289022739=17, srv656230049=80, srv188389411=43, srv2085781652=57, srv234000162=60, srv396762648=69, srv394011465=68, srv769528735=89, srv2055645899=56, srv323821709=64, srv663175520=81, srv101724606=1, srv1728200542=35, srv233799912=59, srv1078594524=5, srv1818987872=40, srv1048792229=3, srv423309836=72, srv744728030=85, srv378682709=67, srv374446587=66, srv416494389=71, srv1232472006=14, srv78818348=91, srv1363386838=20, srv788122441=90, srv913474323=98, srv2045300072=55, srv268119056=61, srv766743637=87, srv118934748=8, srv809924504=92, srv1913479682=44, srv2008676383=51, srv407770218=70, srv865664472=96, srv1058339753=4, srv1426556588=23, srv572474164=77, srv1506040424=29, srv1833282407=41, srv1937204240=46, srv1937078817=45, srv1214471393=10, srv1977648513=48, srv2125508707=58, srv1590130185=30, srv1274829392=16, srv1488949678=27, srv553610373=76, srv663880676=82, srv1816435330=39, srv1970368725=47, srv1211679586=9, srv302970796=63, srv632295599=79, srv734017783=84, srv478931098=74, srv1217808268=11, srv769260112=88, srv1752399437=36, srv2033271374=54, srv1137241657=7, srv1232139054=13, srv833026450=94, srv1024671389=2, srv1775275062=37, srv85673404=95, srv1790529274=38, srv1847645502=42, srv941741303=99, srv744931667=86, srv488530593=75, srv1500347950=28, srv1224119572=12, srv822795415=93, srv2017481343=52, srv1242590913=15, srv1323890131=18, srv1454086126=25, srv1480181004=26, srv1595071430=32, srv1621683733=33, srv1436563639=24, srv1996668886=50, srv670847770=83, srv587121822=78, srv1375568840=21, srv1716094262=34, srv1986652841=49, srv1592365101=31, srv902598655=97, srv452331273=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-08 19:54:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-08 19:54:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-08 19:54:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-08 19:54:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-08 19:54:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-08 19:54:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-08 19:54:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-08 19:54:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-08 19:54:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-08 19:54:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-08 19:54:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-08 19:54:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-08 19:54:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-08 19:54:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-08 19:54:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-08 19:54:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-08 19:54:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-08 19:54:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-08 19:54:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-08 19:54:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-08 19:54:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-08 19:54:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-08 19:54:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-08 19:54:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-08 19:54:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-08 19:54:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-08 19:54:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-08 19:54:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-08 19:54:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-08 19:54:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-08 19:54:17,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-08 19:54:17,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-08 19:54:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-08 19:54:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-08 19:54:17,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-08 19:54:17,300 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039036596193499765, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051203254666475594, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039036596193499765, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16941666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23875, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-08 19:54:17,300 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-08 19:54:17,300 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17443442722582914, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039036596193499765, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051203254666475594, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039036596193499765, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16941666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23875, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-08 19:54:17,624 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 490 ms to try 5000 different iterations.  Found a solution that moves 2656 regions; Going from a computed imbalance of 0.17443442722582914 to a new imbalance of 0.014165696732631778. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020034264201923734); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029731307022799226, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2951111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020034264201923734); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04791666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-08 19:54:17,634 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv374446587:77 , srv1816435330:80 , srv378682709:80 , srv766743637:80 , srv1024671389:81 , srv118934748:81 , srv1048792229:82 , srv1500347950:82 , srv234000162:82 , srv744931667:82 , srv833026450:82 , srv1007581474:83 , srv1232472006:83 , srv188389411:83 , srv1977648513:83 , srv268119056:83 , srv407770218:83 , srv670847770:83 , srv1058339753:84 , srv1093796022:84 , srv1137241657:84 , srv1211679586:84 , srv1289022739:84 , srv1323890131:84 , srv1480181004:84 , srv1506040424:84 , srv1937204240:84 , srv2008676383:84 , srv2085781652:84 , srv452331273:84 , srv1595071430:85 , srv1775275062:85 , srv1847645502:85 , srv2017481343:85 , srv2045300072:85 , srv394011465:85 , srv488530593:85 , srv553610373:85 , srv769528735:85 , srv902598655:85 , srv101724606:86 , srv1242590913:86 , srv1363386838:86 , srv1436563639:86 , srv1454086126:86 , srv1590130185:86 , srv1592365101:86 , srv1790529274:86 , srv1996668886:86 , srv302970796:86 , srv339875449:86 , srv416494389:86 , srv423309836:86 , srv478931098:86 , srv587121822:86 , srv663880676:86 , srv769260112:86 , srv78818348:86 , srv1274829392:87 , srv1426556588:87 , srv1488949678:87 , srv1621683733:87 , srv1716094262:87 , srv1728200542:87 , srv1752399437:87 , srv1833282407:87 , srv1913479682:87 , srv1937078817:87 , srv1970368725:87 , srv2055645899:87 , srv233799912:87 , srv572474164:87 , srv734017783:87 , srv788122441:87 , srv809924504:87 , srv941741303:87 , srv1078594524:88 , srv1214471393:88 , srv1217808268:88 , srv1224119572:88 , srv1232139054:88 , srv1341245919:88 , srv1375568840:88 , srv1426451057:88 , srv2033271374:88 , srv632295599:88 , srv663175520:88 , srv865664472:88 , srv913474323:88 , srv1818987872:89 , srv284121269:89 , srv656230049:89 , srv85673404:89 , srv2031118968:90 , srv323821709:90 , srv744728030:90 , srv1986652841:91 , srv396762648:91 , srv822795415:91 , srv2125508707:515 }
2023-03-08 19:54:17,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-08 19:54:17,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1426451057=22, srv339875449=65, srv284121269=62, srv1341245919=19, srv2031118968=53, srv1007581474=0, srv1093796022=6, srv1289022739=17, srv656230049=80, srv188389411=43, srv2085781652=57, srv234000162=60, srv396762648=69, srv394011465=68, srv769528735=89, srv2055645899=56, srv323821709=64, srv663175520=81, srv101724606=1, srv1728200542=35, srv233799912=59, srv1078594524=5, srv1818987872=40, srv1048792229=3, srv423309836=72, srv744728030=85, srv378682709=67, srv374446587=66, srv416494389=71, srv1232472006=14, srv78818348=91, srv1363386838=20, srv788122441=90, srv913474323=98, srv2045300072=55, srv268119056=61, srv766743637=87, srv118934748=8, srv809924504=92, srv1913479682=44, srv2008676383=51, srv407770218=70, srv865664472=96, srv1058339753=4, srv1426556588=23, srv572474164=77, srv1506040424=29, srv1833282407=41, srv1937204240=46, srv1937078817=45, srv1214471393=10, srv1977648513=48, srv2125508707=58, srv1590130185=30, srv1274829392=16, srv1488949678=27, srv553610373=76, srv663880676=82, srv1816435330=39, srv1970368725=47, srv1211679586=9, srv302970796=63, srv632295599=79, srv734017783=84, srv478931098=74, srv1217808268=11, srv769260112=88, srv1752399437=36, srv2033271374=54, srv1137241657=7, srv1232139054=13, srv833026450=94, srv1024671389=2, srv1775275062=37, srv85673404=95, srv1790529274=38, srv1847645502=42, srv941741303=99, srv744931667=86, srv488530593=75, srv1500347950=28, srv1224119572=12, srv822795415=93, srv2017481343=52, srv1242590913=15, srv1323890131=18, srv1454086126=25, srv1480181004=26, srv1595071430=32, srv1621683733=33, srv1436563639=24, srv1996668886=50, srv670847770=83, srv587121822=78, srv1375568840=21, srv1716094262=34, srv1986652841=49, srv1592365101=31, srv902598655=97, srv452331273=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-08 19:54:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-08 19:54:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-08 19:54:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-08 19:54:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-08 19:54:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-08 19:54:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-08 19:54:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-08 19:54:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-08 19:54:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-08 19:54:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-08 19:54:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-08 19:54:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-08 19:54:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-08 19:54:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-08 19:54:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-08 19:54:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-08 19:54:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-08 19:54:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-08 19:54:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-08 19:54:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-08 19:54:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-08 19:54:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-08 19:54:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-08 19:54:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-08 19:54:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-08 19:54:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-08 19:54:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-08 19:54:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-08 19:54:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-08 19:54:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-08 19:54:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-08 19:54:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-08 19:54:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-08 19:54:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-08 19:54:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-08 19:54:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-08 19:54:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-08 19:54:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-08 19:54:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-08 19:54:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-08 19:54:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-08 19:54:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-08 19:54:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-08 19:54:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-08 19:54:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-08 19:54:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-08 19:54:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-08 19:54:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-08 19:54:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-08 19:54:17,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-08 19:54:17,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-08 19:54:17,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-08 19:54:17,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-08 19:54:17,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-08 19:54:17,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-08 19:54:17,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-08 19:54:17,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-08 19:54:17,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-08 19:54:17,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-08 19:54:17,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-08 19:54:17,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-08 19:54:17,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-08 19:54:17,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-08 19:54:17,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-08 19:54:17,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-08 19:54:17,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-08 19:54:17,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-08 19:54:17,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-08 19:54:17,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-08 19:54:17,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-08 19:54:17,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-08 19:54:17,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-08 19:54:17,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-08 19:54:17,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-08 19:54:17,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-08 19:54:17,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-08 19:54:17,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-08 19:54:17,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-08 19:54:17,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-08 19:54:17,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-08 19:54:17,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-08 19:54:17,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-08 19:54:17,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-08 19:54:17,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-08 19:54:17,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-08 19:54:17,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-08 19:54:17,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-08 19:54:17,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-08 19:54:17,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-08 19:54:17,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-08 19:54:17,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-08 19:54:17,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-08 19:54:17,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-08 19:54:17,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-08 19:54:17,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-08 19:54:17,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-08 19:54:17,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-08 19:54:17,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-08 19:54:17,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-08 19:54:17,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-08 19:54:17,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-08 19:54:17,879 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020034264201923734); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029731307022799226, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020034264201923734); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04791666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-08 19:54:17,879 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-08 19:54:17,880 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014147099023134962, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020034264201923734); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029731307022799226, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020034264201923734); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04791666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-08 19:54:18,180 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 473 ms to try 5000 different iterations.  Found a solution that moves 1030 regions; Going from a computed imbalance of 0.014147099023134962 to a new imbalance of 7.635835268816485E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00399525844422662); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005422652739987697); MoveCostFunction : (multiplier=7.0, imbalance=0.11444444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00399525844422662); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-08 19:54:18,185 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1232139054:82 , srv822795415:82 , srv1986652841:83 , srv284121269:83 , srv396762648:83 , srv833026450:83 , srv865664472:83 , srv1048792229:84 , srv1217808268:84 , srv1232472006:84 , srv1363386838:84 , srv1436563639:84 , srv1454086126:84 , srv1480181004:84 , srv1488949678:84 , srv1592365101:84 , srv1595071430:84 , srv1790529274:84 , srv1833282407:84 , srv1913479682:84 , srv1937078817:84 , srv1996668886:84 , srv2017481343:84 , srv2031118968:84 , srv233799912:84 , srv374446587:84 , srv416494389:84 , srv478931098:84 , srv488530593:84 , srv663880676:84 , srv101724606:85 , srv118934748:85 , srv1214471393:85 , srv1224119572:85 , srv1323890131:85 , srv1375568840:85 , srv1426556588:85 , srv1621683733:85 , srv1752399437:85 , srv1818987872:85 , srv1937204240:85 , srv2008676383:85 , srv2033271374:85 , srv2045300072:85 , srv323821709:85 , srv339875449:85 , srv407770218:85 , srv572474164:85 , srv632295599:85 , srv744931667:85 , srv766743637:85 , srv769260112:85 , srv788122441:85 , srv78818348:85 , srv85673404:85 , srv902598655:85 , srv1078594524:86 , srv1426451057:86 , srv1506040424:86 , srv1775275062:86 , srv1816435330:86 , srv1847645502:86 , srv1970368725:86 , srv302970796:86 , srv394011465:86 , srv423309836:86 , srv553610373:86 , srv656230049:86 , srv663175520:86 , srv734017783:86 , srv809924504:86 , srv913474323:86 , srv941741303:86 , srv1058339753:87 , srv1093796022:87 , srv1137241657:87 , srv1242590913:87 , srv1274829392:87 , srv1341245919:87 , srv1500347950:87 , srv1590130185:87 , srv1716094262:87 , srv1728200542:87 , srv2085781652:87 , srv268119056:87 , srv452331273:87 , srv587121822:87 , srv670847770:87 , srv744728030:87 , srv769528735:87 , srv1024671389:88 , srv1211679586:88 , srv1289022739:88 , srv2055645899:88 , srv234000162:88 , srv1007581474:89 , srv1977648513:89 , srv378682709:89 , srv188389411:90 , srv2125508707:541 }
2023-03-08 19:54:18,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-08 19:54:18,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1426451057=22, srv339875449=65, srv284121269=62, srv1341245919=19, srv2031118968=53, srv1007581474=0, srv1093796022=6, srv1289022739=17, srv656230049=80, srv188389411=43, srv2085781652=57, srv234000162=60, srv396762648=69, srv394011465=68, srv769528735=89, srv2055645899=56, srv323821709=64, srv663175520=81, srv101724606=1, srv1728200542=35, srv233799912=59, srv1078594524=5, srv1818987872=40, srv1048792229=3, srv423309836=72, srv744728030=85, srv378682709=67, srv374446587=66, srv416494389=71, srv1232472006=14, srv78818348=91, srv1363386838=20, srv788122441=90, srv913474323=98, srv2045300072=55, srv268119056=61, srv766743637=87, srv118934748=8, srv809924504=92, srv1913479682=44, srv2008676383=51, srv407770218=70, srv865664472=96, srv1058339753=4, srv1426556588=23, srv572474164=77, srv1506040424=29, srv1833282407=41, srv1937204240=46, srv1937078817=45, srv1214471393=10, srv1977648513=48, srv2125508707=58, srv1590130185=30, srv1274829392=16, srv1488949678=27, srv553610373=76, srv663880676=82, srv1816435330=39, srv1970368725=47, srv1211679586=9, srv302970796=63, srv632295599=79, srv734017783=84, srv478931098=74, srv1217808268=11, srv769260112=88, srv1752399437=36, srv2033271374=54, srv1137241657=7, srv1232139054=13, srv833026450=94, srv1024671389=2, srv1775275062=37, srv85673404=95, srv1790529274=38, srv1847645502=42, srv941741303=99, srv744931667=86, srv488530593=75, srv1500347950=28, srv1224119572=12, srv822795415=93, srv2017481343=52, srv1242590913=15, srv1323890131=18, srv1454086126=25, srv1480181004=26, srv1595071430=32, srv1621683733=33, srv1436563639=24, srv1996668886=50, srv670847770=83, srv587121822=78, srv1375568840=21, srv1716094262=34, srv1986652841=49, srv1592365101=31, srv902598655=97, srv452331273=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-08 19:54:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-08 19:54:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-08 19:54:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-08 19:54:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-08 19:54:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-08 19:54:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-08 19:54:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-08 19:54:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-08 19:54:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-08 19:54:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-08 19:54:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-08 19:54:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-08 19:54:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-08 19:54:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-08 19:54:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-08 19:54:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-08 19:54:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-08 19:54:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-08 19:54:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-08 19:54:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-08 19:54:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-08 19:54:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-08 19:54:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-08 19:54:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-08 19:54:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-08 19:54:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-08 19:54:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-08 19:54:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-08 19:54:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-08 19:54:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-08 19:54:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-08 19:54:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-08 19:54:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-08 19:54:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-08 19:54:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-08 19:54:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-08 19:54:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-08 19:54:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-08 19:54:18,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-08 19:54:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-08 19:54:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-08 19:54:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-08 19:54:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-08 19:54:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-08 19:54:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-08 19:54:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-08 19:54:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-08 19:54:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-08 19:54:18,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-08 19:54:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-08 19:54:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-08 19:54:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-08 19:54:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-08 19:54:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-08 19:54:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-08 19:54:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-08 19:54:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-08 19:54:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-08 19:54:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-08 19:54:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-08 19:54:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-08 19:54:18,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-08 19:54:18,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-08 19:54:18,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-08 19:54:18,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-08 19:54:18,326 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00399525844422662); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005422652739987697); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00399525844422662); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-08 19:54:18,326 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-08 19:54:18,326 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.563713127319045E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00399525844422662); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005422652739987697); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00399525844422662); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-08 19:54:18,678 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 452 ms to try 5000 different iterations.  Found a solution that moves 247 regions; Going from a computed imbalance of 7.563713127319045E-4 to a new imbalance of 1.7493022767383864E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.027444444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-08 19:54:18,680 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv744728030:82 , srv1274829392:83 , srv1289022739:83 , srv1775275062:83 , srv188389411:83 , srv268119056:83 , srv769528735:83 , srv1007581474:84 , srv101724606:84 , srv1024671389:84 , srv1058339753:84 , srv1137241657:84 , srv1211679586:84 , srv1214471393:84 , srv1217808268:84 , srv1242590913:84 , srv1323890131:84 , srv1341245919:84 , srv1375568840:84 , srv1426451057:84 , srv1426556588:84 , srv1488949678:84 , srv1500347950:84 , srv1590130185:84 , srv1595071430:84 , srv1716094262:84 , srv1728200542:84 , srv1752399437:84 , srv1818987872:84 , srv1847645502:84 , srv1970368725:84 , srv1977648513:84 , srv1996668886:84 , srv2017481343:84 , srv2033271374:84 , srv2045300072:84 , srv2055645899:84 , srv234000162:84 , srv284121269:84 , srv302970796:84 , srv378682709:84 , srv407770218:84 , srv553610373:84 , srv572474164:84 , srv587121822:84 , srv632295599:84 , srv656230049:84 , srv663175520:84 , srv663880676:84 , srv670847770:84 , srv744931667:84 , srv766743637:84 , srv769260112:84 , srv788122441:84 , srv78818348:84 , srv809924504:84 , srv833026450:84 , srv865664472:84 , srv913474323:84 , srv1048792229:85 , srv1078594524:85 , srv1093796022:85 , srv118934748:85 , srv1224119572:85 , srv1232139054:85 , srv1232472006:85 , srv1363386838:85 , srv1454086126:85 , srv1480181004:85 , srv1592365101:85 , srv1621683733:85 , srv1790529274:85 , srv1816435330:85 , srv1833282407:85 , srv1937078817:85 , srv1937204240:85 , srv1986652841:85 , srv2008676383:85 , srv2031118968:85 , srv2085781652:85 , srv233799912:85 , srv323821709:85 , srv339875449:85 , srv374446587:85 , srv394011465:85 , srv396762648:85 , srv416494389:85 , srv423309836:85 , srv452331273:85 , srv478931098:85 , srv488530593:85 , srv734017783:85 , srv822795415:85 , srv85673404:85 , srv902598655:85 , srv941741303:85 , srv1436563639:86 , srv1506040424:86 , srv1913479682:86 , srv2125508707:649 }
2023-03-08 19:54:18,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-08 19:54:18,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1426451057=22, srv339875449=65, srv284121269=62, srv1341245919=19, srv2031118968=53, srv1007581474=0, srv1093796022=6, srv1289022739=17, srv656230049=80, srv188389411=43, srv2085781652=57, srv234000162=60, srv396762648=69, srv394011465=68, srv769528735=89, srv2055645899=56, srv323821709=64, srv663175520=81, srv101724606=1, srv1728200542=35, srv233799912=59, srv1078594524=5, srv1818987872=40, srv1048792229=3, srv423309836=72, srv744728030=85, srv378682709=67, srv374446587=66, srv416494389=71, srv1232472006=14, srv78818348=91, srv1363386838=20, srv788122441=90, srv913474323=98, srv2045300072=55, srv268119056=61, srv766743637=87, srv118934748=8, srv809924504=92, srv1913479682=44, srv2008676383=51, srv407770218=70, srv865664472=96, srv1058339753=4, srv1426556588=23, srv572474164=77, srv1506040424=29, srv1833282407=41, srv1937204240=46, srv1937078817=45, srv1214471393=10, srv1977648513=48, srv2125508707=58, srv1590130185=30, srv1274829392=16, srv1488949678=27, srv553610373=76, srv663880676=82, srv1816435330=39, srv1970368725=47, srv1211679586=9, srv302970796=63, srv632295599=79, srv734017783=84, srv478931098=74, srv1217808268=11, srv769260112=88, srv1752399437=36, srv2033271374=54, srv1137241657=7, srv1232139054=13, srv833026450=94, srv1024671389=2, srv1775275062=37, srv85673404=95, srv1790529274=38, srv1847645502=42, srv941741303=99, srv744931667=86, srv488530593=75, srv1500347950=28, srv1224119572=12, srv822795415=93, srv2017481343=52, srv1242590913=15, srv1323890131=18, srv1454086126=25, srv1480181004=26, srv1595071430=32, srv1621683733=33, srv1436563639=24, srv1996668886=50, srv670847770=83, srv587121822=78, srv1375568840=21, srv1716094262=34, srv1986652841=49, srv1592365101=31, srv902598655=97, srv452331273=73} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-08 19:54:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-08 19:54:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-08 19:54:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-08 19:54:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-08 19:54:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-08 19:54:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-08 19:54:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-08 19:54:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-08 19:54:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-08 19:54:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-08 19:54:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-08 19:54:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-08 19:54:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-08 19:54:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-08 19:54:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-08 19:54:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-08 19:54:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-08 19:54:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-08 19:54:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-08 19:54:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-08 19:54:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-08 19:54:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-08 19:54:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-08 19:54:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-08 19:54:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-08 19:54:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-08 19:54:18,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-08 19:54:18,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-08 19:54:18,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-08 19:54:18,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-08 19:54:18,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-08 19:54:18,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-08 19:54:18,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-08 19:54:18,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-08 19:54:18,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-08 19:54:18,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-08 19:54:18,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-08 19:54:18,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-08 19:54:18,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-08 19:54:18,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-08 19:54:18,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-08 19:54:18,826 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.57634918013773E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-08 19:54:18,826 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv744728030:82 , srv1274829392:83 , srv1289022739:83 , srv1775275062:83 , srv188389411:83 , srv268119056:83 , srv769528735:83 , srv1007581474:84 , srv101724606:84 , srv1024671389:84 , srv1058339753:84 , srv1137241657:84 , srv1211679586:84 , srv1214471393:84 , srv1217808268:84 , srv1242590913:84 , srv1323890131:84 , srv1341245919:84 , srv1375568840:84 , srv1426451057:84 , srv1426556588:84 , srv1488949678:84 , srv1500347950:84 , srv1590130185:84 , srv1595071430:84 , srv1716094262:84 , srv1728200542:84 , srv1752399437:84 , srv1818987872:84 , srv1847645502:84 , srv1970368725:84 , srv1977648513:84 , srv1996668886:84 , srv2017481343:84 , srv2033271374:84 , srv2045300072:84 , srv2055645899:84 , srv234000162:84 , srv284121269:84 , srv302970796:84 , srv378682709:84 , srv407770218:84 , srv553610373:84 , srv572474164:84 , srv587121822:84 , srv632295599:84 , srv656230049:84 , srv663175520:84 , srv663880676:84 , srv670847770:84 , srv744931667:84 , srv766743637:84 , srv769260112:84 , srv788122441:84 , srv78818348:84 , srv809924504:84 , srv833026450:84 , srv865664472:84 , srv913474323:84 , srv1048792229:85 , srv1078594524:85 , srv1093796022:85 , srv118934748:85 , srv1224119572:85 , srv1232139054:85 , srv1232472006:85 , srv1363386838:85 , srv1454086126:85 , srv1480181004:85 , srv1592365101:85 , srv1621683733:85 , srv1790529274:85 , srv1816435330:85 , srv1833282407:85 , srv1937078817:85 , srv1937204240:85 , srv1986652841:85 , srv2008676383:85 , srv2031118968:85 , srv2085781652:85 , srv233799912:85 , srv323821709:85 , srv339875449:85 , srv374446587:85 , srv394011465:85 , srv396762648:85 , srv416494389:85 , srv423309836:85 , srv452331273:85 , srv478931098:85 , srv488530593:85 , srv734017783:85 , srv822795415:85 , srv85673404:85 , srv902598655:85 , srv941741303:85 , srv1436563639:86 , srv1506040424:86 , srv1913479682:86 , srv2125508707:649 }
2023-03-08 19:54:18,856 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=219 (was 219), ProcessCount=170 (was 170), AvailableMemoryMB=8481 (was 8980)
2023-03-08 19:54:18,866 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=219, ProcessCount=170, AvailableMemoryMB=8483
2023-03-08 19:54:18,867 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-08 19:54:18,867 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-08 19:54:18,875 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1007990124:84 , srv1106587975:84 , srv1382791777:84 , srv1385262739:84 , srv1420970503:84 , srv1473994920:84 , srv1481759733:84 , srv1524325873:84 , srv1612599140:84 , srv1638487272:84 , srv1734195291:84 , srv175180186:84 , srv1756155934:84 , srv18194103:84 , srv1875597166:84 , srv1876060231:84 , srv1901471577:84 , srv1991131676:84 , srv2078659392:84 , srv296586401:84 , srv330223850:84 , srv397567185:84 , srv455462414:84 , srv548002824:84 , srv571620742:84 , srv612559276:84 , srv797156506:84 , srv848985512:84 , srv869643006:84 , srv1988568072:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-08 19:54:18,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-08 19:54:18,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1524325873=7, srv548002824=24, srv1988568072=17, srv1734195291=10, srv848985512=28, srv797156506=27, srv1473994920=5, srv397567185=22, srv18194103=13, srv869643006=29, srv1756155934=12, srv1382791777=2, srv1875597166=14, srv455462414=23, srv1420970503=4, srv571620742=25, srv1385262739=3, srv1481759733=6, srv1612599140=8, srv330223850=21, srv612559276=26, srv1876060231=15, srv1007990124=0, srv1106587975=1, srv175180186=11, srv296586401=20, srv1901471577=16, srv1638487272=9, srv1991131676=18, srv2078659392=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-08 19:54:18,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-08 19:54:18,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-08 19:54:18,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-08 19:54:18,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-08 19:54:18,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-08 19:54:18,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-08 19:54:18,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-08 19:54:18,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-08 19:54:18,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-08 19:54:18,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-08 19:54:18,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-08 19:54:18,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-08 19:54:18,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-08 19:54:18,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-08 19:54:18,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-08 19:54:18,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-08 19:54:18,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-08 19:54:18,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-08 19:54:18,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-08 19:54:18,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-08 19:54:18,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-08 19:54:18,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-08 19:54:18,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-08 19:54:18,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-08 19:54:18,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-08 19:54:18,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-08 19:54:18,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-08 19:54:18,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-08 19:54:18,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-08 19:54:18,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-08 19:54:18,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-08 19:54:18,902 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1529183630879944, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-08 19:54:18,902 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912674884033125, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1529183630879944, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-08 19:56:18,890 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6113902 different iterations.  Found a solution that moves 1869 regions; Going from a computed imbalance of 0.9912674884033125 to a new imbalance of 4.362339238146111E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6922222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-08 19:56:18,896 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1007990124:90 , srv1106587975:90 , srv1382791777:90 , srv1385262739:90 , srv1420970503:90 , srv1473994920:90 , srv1481759733:90 , srv1524325873:90 , srv1612599140:90 , srv1638487272:90 , srv1734195291:90 , srv175180186:90 , srv1756155934:90 , srv18194103:90 , srv1875597166:90 , srv1876060231:90 , srv1901471577:90 , srv1988568072:90 , srv1991131676:90 , srv2078659392:90 , srv296586401:90 , srv330223850:90 , srv397567185:90 , srv455462414:90 , srv548002824:90 , srv571620742:90 , srv612559276:90 , srv797156506:90 , srv848985512:90 , srv869643006:90 }
2023-03-08 19:56:18,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-08 19:56:18,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1524325873=7, srv548002824=24, srv1988568072=17, srv1734195291=10, srv848985512=28, srv797156506=27, srv1473994920=5, srv397567185=22, srv18194103=13, srv869643006=29, srv1756155934=12, srv1382791777=2, srv1875597166=14, srv455462414=23, srv1420970503=4, srv571620742=25, srv1385262739=3, srv1481759733=6, srv1612599140=8, srv330223850=21, srv612559276=26, srv1876060231=15, srv1007990124=0, srv1106587975=1, srv175180186=11, srv296586401=20, srv1901471577=16, srv1638487272=9, srv1991131676=18, srv2078659392=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-08 19:56:18,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-08 19:56:18,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-08 19:56:18,914 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-08 19:56:18,914 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1007990124:90 , srv1106587975:90 , srv1382791777:90 , srv1385262739:90 , srv1420970503:90 , srv1473994920:90 , srv1481759733:90 , srv1524325873:90 , srv1612599140:90 , srv1638487272:90 , srv1734195291:90 , srv175180186:90 , srv1756155934:90 , srv18194103:90 , srv1875597166:90 , srv1876060231:90 , srv1901471577:90 , srv1988568072:90 , srv1991131676:90 , srv2078659392:90 , srv296586401:90 , srv330223850:90 , srv397567185:90 , srv455462414:90 , srv548002824:90 , srv571620742:90 , srv612559276:90 , srv797156506:90 , srv848985512:90 , srv869643006:90 }
2023-03-08 19:56:18,925 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=173 (was 219), ProcessCount=170 (was 170), AvailableMemoryMB=8105 (was 8483)