2023-03-03 06:54:01,751 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-03 06:54:02,125 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-03 06:54:02,237 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-03 06:54:02,294 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-03 06:54:02,301 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-03 06:54:02,314 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=451, ProcessCount=171, AvailableMemoryMB=6694
2023-03-03 06:54:02,316 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-03 06:54:02,316 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-03 06:54:02,450 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004763325:84 , srv1004801932:84 , srv1024953478:84 , srv1041171784:84 , srv1057234407:84 , srv1071140816:84 , srv109737493:84 , srv1098743837:84 , srv1127273051:84 , srv1131591638:84 , srv1134447516:84 , srv1158685569:84 , srv116104166:84 , srv117255589:84 , srv1206052547:84 , srv1207662565:84 , srv1227980976:84 , srv1231575609:84 , srv1232893532:84 , srv1295615919:84 , srv1310585819:84 , srv1324584762:84 , srv1367137635:84 , srv1368782436:84 , srv1373104488:84 , srv1385968174:84 , srv1405074165:84 , srv1408118882:84 , srv1414193898:84 , srv1421926921:84 , srv1469852211:84 , srv1507114008:84 , srv1544681992:84 , srv1569585196:84 , srv1594236782:84 , srv1599826251:84 , srv1605466460:84 , srv1609527369:84 , srv1680456669:84 , srv1683852706:84 , srv1700683277:84 , srv1748036411:84 , srv1752050456:84 , srv1753073235:84 , srv176145038:84 , srv1763139225:84 , srv1766555821:84 , srv1843369259:84 , srv1850308486:84 , srv1851531900:84 , srv1874556138:84 , srv1886165001:84 , srv1921127106:84 , srv1954097889:84 , srv1967038364:84 , srv1971939143:84 , srv2045248281:84 , srv2091906523:84 , srv2101097745:84 , srv2142762510:84 , srv218596602:84 , srv23529383:84 , srv236488901:84 , srv262356220:84 , srv26664082:84 , srv283068948:84 , srv307338312:84 , srv31678768:84 , srv335378382:84 , srv415505413:84 , srv466630775:84 , srv478355009:84 , srv517222357:84 , srv54573070:84 , srv547115652:84 , srv551670353:84 , srv558225910:84 , srv561996094:84 , srv567756214:84 , srv573363651:84 , srv577631723:84 , srv599252224:84 , srv650308076:84 , srv65064659:84 , srv662634366:84 , srv668531860:84 , srv693635180:84 , srv720257502:84 , srv724085831:84 , srv737969943:84 , srv743118374:84 , srv775357429:84 , srv806531572:84 , srv807752385:84 , srv814357057:84 , srv889739434:84 , srv909772922:84 , srv919549501:84 , srv978150082:84 , srv2015613202:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-03 06:54:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-03 06:54:02,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1131591638=9, srv668531860=86, srv54573070=74, srv2142762510=60, srv919549501=98, srv1158685569=11, srv23529383=62, srv415505413=70, srv1231575609=17, srv1683852706=39, srv1469852211=30, srv1041171784=3, srv573363651=80, srv737969943=90, srv109737493=6, srv909772922=97, srv724085831=89, srv1971939143=55, srv1134447516=10, srv31678768=68, srv1408118882=27, srv1127273051=8, srv2015613202=56, srv547115652=75, srv889739434=96, srv65064659=84, srv551670353=76, srv1700683277=40, srv1569585196=33, srv1057234407=4, srv650308076=83, srv1004801932=1, srv1232893532=18, srv558225910=77, srv335378382=69, srv117255589=13, srv307338312=67, srv1405074165=26, srv1850308486=48, srv218596602=61, srv262356220=64, srv1921127106=52, srv1004763325=0, srv1367137635=22, srv1507114008=31, srv1766555821=46, srv693635180=87, srv1874556138=50, srv978150082=99, srv1024953478=2, srv176145038=44, srv116104166=12, srv1605466460=36, srv1227980976=16, srv775357429=92, srv1098743837=7, srv1310585819=20, srv1763139225=45, srv1207662565=15, srv1385968174=25, srv743118374=91, srv2091906523=58, srv567756214=79, srv1414193898=28, srv1599826251=35, srv1967038364=54, srv478355009=72, srv1753073235=43, srv577631723=81, srv720257502=88, srv1851531900=49, srv1544681992=32, srv236488901=63, srv1421926921=29, srv561996094=78, srv1886165001=51, srv466630775=71, srv1752050456=42, srv814357057=95, srv517222357=73, srv1609527369=37, srv807752385=94, srv1071140816=5, srv1748036411=41, srv1594236782=34, srv26664082=65, srv1324584762=21, srv806531572=93, srv2045248281=57, srv1680456669=38, srv1368782436=23, srv283068948=66, srv1954097889=53, srv1843369259=47, srv2101097745=59, srv662634366=85, srv599252224=82, srv1206052547=14, srv1295615919=19, srv1373104488=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-03 06:54:02,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-03 06:54:02,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-03 06:54:02,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-03 06:54:02,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-03 06:54:02,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-03 06:54:02,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-03 06:54:02,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-03 06:54:02,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-03 06:54:02,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-03 06:54:02,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-03 06:54:02,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-03 06:54:02,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-03 06:54:02,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-03 06:54:02,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-03 06:54:02,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-03 06:54:02,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-03 06:54:02,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-03 06:54:02,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-03 06:54:02,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-03 06:54:02,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-03 06:54:02,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-03 06:54:02,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-03 06:54:02,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-03 06:54:02,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-03 06:54:02,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-03 06:54:02,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-03 06:54:02,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-03 06:54:02,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-03 06:54:02,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-03 06:54:02,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-03 06:54:02,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-03 06:54:02,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-03 06:54:02,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-03 06:54:02,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-03 06:54:02,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-03 06:54:02,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-03 06:54:02,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-03 06:54:02,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-03 06:54:02,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-03 06:54:02,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-03 06:54:02,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-03 06:54:02,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-03 06:54:02,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-03 06:54:02,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-03 06:54:02,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-03 06:54:02,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-03 06:54:02,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-03 06:54:02,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-03 06:54:02,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-03 06:54:02,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-03 06:54:02,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-03 06:54:02,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-03 06:54:02,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-03 06:54:02,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-03 06:54:02,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-03 06:54:02,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-03 06:54:02,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-03 06:54:02,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-03 06:54:02,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-03 06:54:02,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-03 06:54:02,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-03 06:54:02,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-03 06:54:02,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-03 06:54:02,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-03 06:54:02,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-03 06:54:02,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-03 06:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-03 06:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-03 06:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-03 06:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-03 06:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-03 06:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-03 06:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-03 06:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-03 06:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-03 06:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-03 06:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-03 06:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-03 06:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-03 06:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-03 06:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-03 06:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-03 06:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-03 06:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-03 06:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-03 06:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-03 06:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-03 06:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-03 06:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-03 06:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-03 06:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-03 06:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-03 06:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-03 06:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-03 06:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-03 06:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-03 06:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-03 06:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-03 06:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-03 06:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-03 06:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-03 06:54:02,817 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-03 06:54:02,817 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-03 06:54:02,817 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-03 06:54:03,527 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 944 ms to try 5000 different iterations.  Found a solution that moves 4643 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.176840130201909. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03826841882396886, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05590596562213785, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5158888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03826841882396886, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17208333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23825, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-03 06:54:03,554 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv117255589:82 , srv466630775:82 , srv1098743837:83 , srv1127273051:83 , srv1207662565:83 , srv176145038:83 , srv335378382:83 , srv567756214:83 , srv693635180:83 , srv814357057:83 , srv1024953478:84 , srv109737493:84 , srv1324584762:84 , srv1605466460:84 , srv1609527369:84 , srv1700683277:84 , srv1748036411:84 , srv1763139225:84 , srv1851531900:84 , srv1921127106:84 , srv1971939143:84 , srv218596602:84 , srv558225910:84 , srv662634366:84 , srv724085831:84 , srv775357429:84 , srv1295615919:85 , srv1408118882:85 , srv1414193898:85 , srv1544681992:85 , srv1594236782:85 , srv1680456669:85 , srv1753073235:85 , srv1967038364:85 , srv517222357:85 , srv547115652:85 , srv978150082:85 , srv1057234407:86 , srv116104166:86 , srv1310585819:86 , srv1373104488:86 , srv1385968174:86 , srv1569585196:86 , srv1599826251:86 , srv1850308486:86 , srv2101097745:86 , srv283068948:86 , srv307338312:86 , srv65064659:86 , srv889739434:86 , srv1131591638:87 , srv1367137635:87 , srv1368782436:87 , srv1421926921:87 , srv1469852211:87 , srv1507114008:87 , srv1843369259:87 , srv1874556138:87 , srv2045248281:87 , srv2142762510:87 , srv478355009:87 , srv561996094:87 , srv577631723:87 , srv650308076:87 , srv668531860:87 , srv743118374:87 , srv806531572:87 , srv1071140816:88 , srv1206052547:88 , srv1227980976:88 , srv1232893532:88 , srv1683852706:88 , srv23529383:88 , srv26664082:88 , srv31678768:88 , srv551670353:88 , srv737969943:88 , srv909772922:88 , srv1134447516:89 , srv1158685569:89 , srv1231575609:89 , srv1405074165:89 , srv1766555821:89 , srv1954097889:89 , srv262356220:89 , srv599252224:89 , srv807752385:89 , srv919549501:89 , srv1004801932:90 , srv573363651:90 , srv2091906523:91 , srv415505413:91 , srv54573070:91 , srv720257502:91 , srv1004763325:92 , srv1041171784:92 , srv236488901:92 , srv1886165001:93 , srv1752050456:95 , srv2015613202:430 }
2023-03-03 06:54:03,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-03 06:54:03,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1131591638=9, srv668531860=86, srv54573070=74, srv2142762510=60, srv919549501=98, srv1158685569=11, srv23529383=62, srv415505413=70, srv1231575609=17, srv1683852706=39, srv1469852211=30, srv1041171784=3, srv573363651=80, srv737969943=90, srv109737493=6, srv909772922=97, srv724085831=89, srv1971939143=55, srv1134447516=10, srv31678768=68, srv1408118882=27, srv1127273051=8, srv2015613202=56, srv547115652=75, srv889739434=96, srv65064659=84, srv551670353=76, srv1700683277=40, srv1569585196=33, srv1057234407=4, srv650308076=83, srv1004801932=1, srv1232893532=18, srv558225910=77, srv335378382=69, srv117255589=13, srv307338312=67, srv1405074165=26, srv1850308486=48, srv218596602=61, srv262356220=64, srv1921127106=52, srv1004763325=0, srv1367137635=22, srv1507114008=31, srv1766555821=46, srv693635180=87, srv1874556138=50, srv978150082=99, srv1024953478=2, srv176145038=44, srv116104166=12, srv1605466460=36, srv1227980976=16, srv775357429=92, srv1098743837=7, srv1310585819=20, srv1763139225=45, srv1207662565=15, srv1385968174=25, srv743118374=91, srv2091906523=58, srv567756214=79, srv1414193898=28, srv1599826251=35, srv1967038364=54, srv478355009=72, srv1753073235=43, srv577631723=81, srv720257502=88, srv1851531900=49, srv1544681992=32, srv236488901=63, srv1421926921=29, srv561996094=78, srv1886165001=51, srv466630775=71, srv1752050456=42, srv814357057=95, srv517222357=73, srv1609527369=37, srv807752385=94, srv1071140816=5, srv1748036411=41, srv1594236782=34, srv26664082=65, srv1324584762=21, srv806531572=93, srv2045248281=57, srv1680456669=38, srv1368782436=23, srv283068948=66, srv1954097889=53, srv1843369259=47, srv2101097745=59, srv662634366=85, srv599252224=82, srv1206052547=14, srv1295615919=19, srv1373104488=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-03 06:54:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-03 06:54:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-03 06:54:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-03 06:54:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-03 06:54:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-03 06:54:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-03 06:54:03,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-03 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-03 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-03 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-03 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-03 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-03 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-03 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-03 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-03 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-03 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-03 06:54:03,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-03 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-03 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-03 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-03 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-03 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-03 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-03 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-03 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-03 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-03 06:54:03,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-03 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-03 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-03 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-03 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-03 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-03 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-03 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-03 06:54:03,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-03 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-03 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-03 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-03 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-03 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-03 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-03 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-03 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-03 06:54:03,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-03 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-03 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-03 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-03 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-03 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-03 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-03 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-03 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-03 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-03 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-03 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-03 06:54:03,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-03 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-03 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-03 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-03 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-03 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-03 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-03 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-03 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-03 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-03 06:54:03,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-03 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-03 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-03 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-03 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-03 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-03 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-03 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-03 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-03 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-03 06:54:03,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-03 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-03 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-03 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-03 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-03 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-03 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-03 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-03 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-03 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-03 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-03 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-03 06:54:03,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-03 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-03 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-03 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-03 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-03 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-03 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-03 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-03 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-03 06:54:03,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-03 06:54:03,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-03 06:54:03,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-03 06:54:03,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-03 06:54:03,777 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03826841882396886, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05590596562213785, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03826841882396886, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17208333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23825, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-03 06:54:03,778 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-03 06:54:03,778 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17680761922103788, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03826841882396886, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05590596562213785, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03826841882396886, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17208333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23825, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-03 06:54:04,137 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 522 ms to try 5000 different iterations.  Found a solution that moves 2653 regions; Going from a computed imbalance of 0.17680761922103788 to a new imbalance of 0.01508090967491901. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020501871232412608); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034220516814387404, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2947777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020501871232412608); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.047, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-03 06:54:04,149 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1752050456:77 , srv1041171784:79 , srv1886165001:80 , srv236488901:80 , srv1158685569:81 , srv2091906523:81 , srv720257502:81 , srv1134447516:82 , srv1954097889:82 , srv415505413:82 , srv551670353:82 , srv599252224:82 , srv909772922:82 , srv1004763325:83 , srv1004801932:83 , srv1131591638:83 , srv1232893532:83 , srv1368782436:83 , srv1874556138:83 , srv2142762510:83 , srv31678768:83 , srv54573070:83 , srv743118374:83 , srv807752385:83 , srv919549501:83 , srv1507114008:84 , srv1850308486:84 , srv2045248281:84 , srv262356220:84 , srv26664082:84 , srv307338312:84 , srv573363651:84 , srv737969943:84 , srv1405074165:85 , srv1421926921:85 , srv1469852211:85 , srv1569585196:85 , srv1599826251:85 , srv1753073235:85 , srv23529383:85 , srv478355009:85 , srv65064659:85 , srv1071140816:86 , srv109737493:86 , srv116104166:86 , srv1227980976:86 , srv1367137635:86 , srv1373104488:86 , srv1385968174:86 , srv1544681992:86 , srv1683852706:86 , srv1700683277:86 , srv1766555821:86 , srv517222357:86 , srv561996094:86 , srv650308076:86 , srv806531572:86 , srv978150082:86 , srv1024953478:87 , srv1057234407:87 , srv1207662565:87 , srv1231575609:87 , srv1310585819:87 , srv1324584762:87 , srv1408118882:87 , srv1414193898:87 , srv1843369259:87 , srv1921127106:87 , srv1967038364:87 , srv2101097745:87 , srv567756214:87 , srv577631723:87 , srv775357429:87 , srv889739434:87 , srv1098743837:88 , srv1127273051:88 , srv1295615919:88 , srv1594236782:88 , srv1609527369:88 , srv1680456669:88 , srv176145038:88 , srv1763139225:88 , srv1851531900:88 , srv1971939143:88 , srv283068948:88 , srv693635180:88 , srv1206052547:89 , srv466630775:89 , srv547115652:89 , srv558225910:89 , srv668531860:89 , srv724085831:89 , srv814357057:89 , srv117255589:90 , srv218596602:90 , srv335378382:90 , srv662634366:90 , srv1605466460:91 , srv1748036411:92 , srv2015613202:526 }
2023-03-03 06:54:04,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-03 06:54:04,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1131591638=9, srv668531860=86, srv54573070=74, srv2142762510=60, srv919549501=98, srv1158685569=11, srv23529383=62, srv415505413=70, srv1231575609=17, srv1683852706=39, srv1469852211=30, srv1041171784=3, srv573363651=80, srv737969943=90, srv109737493=6, srv909772922=97, srv724085831=89, srv1971939143=55, srv1134447516=10, srv31678768=68, srv1408118882=27, srv1127273051=8, srv2015613202=56, srv547115652=75, srv889739434=96, srv65064659=84, srv551670353=76, srv1700683277=40, srv1569585196=33, srv1057234407=4, srv650308076=83, srv1004801932=1, srv1232893532=18, srv558225910=77, srv335378382=69, srv117255589=13, srv307338312=67, srv1405074165=26, srv1850308486=48, srv218596602=61, srv262356220=64, srv1921127106=52, srv1004763325=0, srv1367137635=22, srv1507114008=31, srv1766555821=46, srv693635180=87, srv1874556138=50, srv978150082=99, srv1024953478=2, srv176145038=44, srv116104166=12, srv1605466460=36, srv1227980976=16, srv775357429=92, srv1098743837=7, srv1310585819=20, srv1763139225=45, srv1207662565=15, srv1385968174=25, srv743118374=91, srv2091906523=58, srv567756214=79, srv1414193898=28, srv1599826251=35, srv1967038364=54, srv478355009=72, srv1753073235=43, srv577631723=81, srv720257502=88, srv1851531900=49, srv1544681992=32, srv236488901=63, srv1421926921=29, srv561996094=78, srv1886165001=51, srv466630775=71, srv1752050456=42, srv814357057=95, srv517222357=73, srv1609527369=37, srv807752385=94, srv1071140816=5, srv1748036411=41, srv1594236782=34, srv26664082=65, srv1324584762=21, srv806531572=93, srv2045248281=57, srv1680456669=38, srv1368782436=23, srv283068948=66, srv1954097889=53, srv1843369259=47, srv2101097745=59, srv662634366=85, srv599252224=82, srv1206052547=14, srv1295615919=19, srv1373104488=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-03 06:54:04,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-03 06:54:04,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-03 06:54:04,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-03 06:54:04,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-03 06:54:04,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-03 06:54:04,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-03 06:54:04,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-03 06:54:04,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-03 06:54:04,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-03 06:54:04,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-03 06:54:04,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-03 06:54:04,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-03 06:54:04,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-03 06:54:04,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-03 06:54:04,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-03 06:54:04,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-03 06:54:04,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-03 06:54:04,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-03 06:54:04,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-03 06:54:04,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-03 06:54:04,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-03 06:54:04,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-03 06:54:04,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-03 06:54:04,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-03 06:54:04,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-03 06:54:04,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-03 06:54:04,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-03 06:54:04,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-03 06:54:04,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-03 06:54:04,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-03 06:54:04,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-03 06:54:04,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-03 06:54:04,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-03 06:54:04,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-03 06:54:04,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-03 06:54:04,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-03 06:54:04,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-03 06:54:04,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-03 06:54:04,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-03 06:54:04,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-03 06:54:04,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-03 06:54:04,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-03 06:54:04,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-03 06:54:04,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-03 06:54:04,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-03 06:54:04,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-03 06:54:04,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-03 06:54:04,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-03 06:54:04,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-03 06:54:04,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-03 06:54:04,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-03 06:54:04,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-03 06:54:04,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-03 06:54:04,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-03 06:54:04,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-03 06:54:04,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-03 06:54:04,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-03 06:54:04,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-03 06:54:04,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-03 06:54:04,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-03 06:54:04,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-03 06:54:04,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-03 06:54:04,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-03 06:54:04,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-03 06:54:04,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-03 06:54:04,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-03 06:54:04,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-03 06:54:04,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-03 06:54:04,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-03 06:54:04,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-03 06:54:04,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-03 06:54:04,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-03 06:54:04,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-03 06:54:04,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-03 06:54:04,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-03 06:54:04,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-03 06:54:04,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-03 06:54:04,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-03 06:54:04,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-03 06:54:04,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-03 06:54:04,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-03 06:54:04,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-03 06:54:04,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-03 06:54:04,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-03 06:54:04,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-03 06:54:04,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-03 06:54:04,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-03 06:54:04,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-03 06:54:04,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-03 06:54:04,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-03 06:54:04,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-03 06:54:04,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-03 06:54:04,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-03 06:54:04,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-03 06:54:04,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-03 06:54:04,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-03 06:54:04,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-03 06:54:04,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-03 06:54:04,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-03 06:54:04,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-03 06:54:04,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-03 06:54:04,335 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020501871232412608); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034220516814387404, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020501871232412608); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.047, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-03 06:54:04,335 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-03 06:54:04,335 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015062332971871175, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020501871232412608); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034220516814387404, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020501871232412608); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.047, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-03 06:54:04,594 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 376 ms to try 5000 different iterations.  Found a solution that moves 1018 regions; Going from a computed imbalance of 0.015062332971871175 to a new imbalance of 8.176409119941912E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006001205349269034); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010304020550550783); MoveCostFunction : (multiplier=7.0, imbalance=0.1131111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006001205349269034); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-03 06:54:04,597 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1206052547:80 , srv668531860:81 , srv1004763325:82 , srv1231575609:82 , srv1748036411:82 , srv1766555821:82 , srv1004801932:83 , srv1071140816:83 , srv1227980976:83 , srv1405074165:83 , srv1594236782:83 , srv1605466460:83 , srv54573070:83 , srv662634366:83 , srv1057234407:84 , srv1295615919:84 , srv1310585819:84 , srv1385968174:84 , srv1680456669:84 , srv1843369259:84 , srv1886165001:84 , srv218596602:84 , srv262356220:84 , srv283068948:84 , srv415505413:84 , srv547115652:84 , srv573363651:84 , srv577631723:84 , srv807752385:84 , srv116104166:85 , srv117255589:85 , srv1367137635:85 , srv1373104488:85 , srv1414193898:85 , srv1421926921:85 , srv1683852706:85 , srv1851531900:85 , srv2101097745:85 , srv23529383:85 , srv236488901:85 , srv26664082:85 , srv335378382:85 , srv478355009:85 , srv558225910:85 , srv561996094:85 , srv724085831:85 , srv806531572:85 , srv1024953478:86 , srv1127273051:86 , srv1134447516:86 , srv1232893532:86 , srv1324584762:86 , srv1408118882:86 , srv1469852211:86 , srv1544681992:86 , srv1569585196:86 , srv1599826251:86 , srv1752050456:86 , srv1763139225:86 , srv1874556138:86 , srv1921127106:86 , srv1967038364:86 , srv1971939143:86 , srv2045248281:86 , srv2091906523:86 , srv466630775:86 , srv599252224:86 , srv650308076:86 , srv65064659:86 , srv693635180:86 , srv720257502:86 , srv737969943:86 , srv814357057:86 , srv889739434:86 , srv919549501:86 , srv978150082:86 , srv109737493:87 , srv1098743837:87 , srv1158685569:87 , srv1207662565:87 , srv1507114008:87 , srv1700683277:87 , srv1753073235:87 , srv1954097889:87 , srv2142762510:87 , srv307338312:87 , srv31678768:87 , srv517222357:87 , srv551670353:87 , srv743118374:87 , srv775357429:87 , srv909772922:87 , srv1041171784:88 , srv1131591638:88 , srv1368782436:88 , srv1609527369:88 , srv176145038:88 , srv1850308486:88 , srv567756214:88 , srv2015613202:555 }
2023-03-03 06:54:04,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-03 06:54:04,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1131591638=9, srv668531860=86, srv54573070=74, srv2142762510=60, srv919549501=98, srv1158685569=11, srv23529383=62, srv415505413=70, srv1231575609=17, srv1683852706=39, srv1469852211=30, srv1041171784=3, srv573363651=80, srv737969943=90, srv109737493=6, srv909772922=97, srv724085831=89, srv1971939143=55, srv1134447516=10, srv31678768=68, srv1408118882=27, srv1127273051=8, srv2015613202=56, srv547115652=75, srv889739434=96, srv65064659=84, srv551670353=76, srv1700683277=40, srv1569585196=33, srv1057234407=4, srv650308076=83, srv1004801932=1, srv1232893532=18, srv558225910=77, srv335378382=69, srv117255589=13, srv307338312=67, srv1405074165=26, srv1850308486=48, srv218596602=61, srv262356220=64, srv1921127106=52, srv1004763325=0, srv1367137635=22, srv1507114008=31, srv1766555821=46, srv693635180=87, srv1874556138=50, srv978150082=99, srv1024953478=2, srv176145038=44, srv116104166=12, srv1605466460=36, srv1227980976=16, srv775357429=92, srv1098743837=7, srv1310585819=20, srv1763139225=45, srv1207662565=15, srv1385968174=25, srv743118374=91, srv2091906523=58, srv567756214=79, srv1414193898=28, srv1599826251=35, srv1967038364=54, srv478355009=72, srv1753073235=43, srv577631723=81, srv720257502=88, srv1851531900=49, srv1544681992=32, srv236488901=63, srv1421926921=29, srv561996094=78, srv1886165001=51, srv466630775=71, srv1752050456=42, srv814357057=95, srv517222357=73, srv1609527369=37, srv807752385=94, srv1071140816=5, srv1748036411=41, srv1594236782=34, srv26664082=65, srv1324584762=21, srv806531572=93, srv2045248281=57, srv1680456669=38, srv1368782436=23, srv283068948=66, srv1954097889=53, srv1843369259=47, srv2101097745=59, srv662634366=85, srv599252224=82, srv1206052547=14, srv1295615919=19, srv1373104488=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-03 06:54:04,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-03 06:54:04,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-03 06:54:04,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-03 06:54:04,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-03 06:54:04,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-03 06:54:04,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-03 06:54:04,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-03 06:54:04,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-03 06:54:04,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-03 06:54:04,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-03 06:54:04,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-03 06:54:04,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-03 06:54:04,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-03 06:54:04,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-03 06:54:04,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-03 06:54:04,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-03 06:54:04,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-03 06:54:04,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-03 06:54:04,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-03 06:54:04,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-03 06:54:04,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-03 06:54:04,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-03 06:54:04,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-03 06:54:04,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-03 06:54:04,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-03 06:54:04,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-03 06:54:04,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-03 06:54:04,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-03 06:54:04,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-03 06:54:04,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-03 06:54:04,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-03 06:54:04,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-03 06:54:04,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-03 06:54:04,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-03 06:54:04,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-03 06:54:04,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-03 06:54:04,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-03 06:54:04,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-03 06:54:04,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-03 06:54:04,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-03 06:54:04,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-03 06:54:04,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-03 06:54:04,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-03 06:54:04,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-03 06:54:04,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-03 06:54:04,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-03 06:54:04,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-03 06:54:04,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-03 06:54:04,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-03 06:54:04,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-03 06:54:04,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-03 06:54:04,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-03 06:54:04,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-03 06:54:04,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-03 06:54:04,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-03 06:54:04,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-03 06:54:04,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-03 06:54:04,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-03 06:54:04,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-03 06:54:04,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-03 06:54:04,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-03 06:54:04,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-03 06:54:04,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-03 06:54:04,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-03 06:54:04,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-03 06:54:04,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-03 06:54:04,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-03 06:54:04,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-03 06:54:04,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-03 06:54:04,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-03 06:54:04,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-03 06:54:04,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-03 06:54:04,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-03 06:54:04,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-03 06:54:04,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-03 06:54:04,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-03 06:54:04,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-03 06:54:04,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-03 06:54:04,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-03 06:54:04,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-03 06:54:04,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-03 06:54:04,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-03 06:54:04,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-03 06:54:04,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-03 06:54:04,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-03 06:54:04,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-03 06:54:04,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-03 06:54:04,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-03 06:54:04,747 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006001205349269034); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010304020550550783); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006001205349269034); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-03 06:54:04,747 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-03 06:54:04,748 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.105127236403666E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006001205349269034); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010304020550550783); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006001205349269034); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-03 06:54:05,067 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 429 ms to try 5000 different iterations.  Found a solution that moves 267 regions; Going from a computed imbalance of 8.105127236403666E-4 to a new imbalance of 2.0316508822778633E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.029666666666666668, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-03 06:54:05,068 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1609527369:82 , srv176145038:82 , srv889739434:82 , srv1041171784:83 , srv1469852211:83 , srv1683852706:83 , srv650308076:83 , srv1004763325:84 , srv1071140816:84 , srv109737493:84 , srv1098743837:84 , srv1131591638:84 , srv1231575609:84 , srv1367137635:84 , srv1368782436:84 , srv1408118882:84 , srv1605466460:84 , srv1748036411:84 , srv1752050456:84 , srv1843369259:84 , srv1850308486:84 , srv1954097889:84 , srv1967038364:84 , srv2101097745:84 , srv218596602:84 , srv23529383:84 , srv283068948:84 , srv31678768:84 , srv335378382:84 , srv517222357:84 , srv547115652:84 , srv558225910:84 , srv561996094:84 , srv567756214:84 , srv573363651:84 , srv577631723:84 , srv693635180:84 , srv720257502:84 , srv737969943:84 , srv775357429:84 , srv806531572:84 , srv814357057:84 , srv919549501:84 , srv1024953478:85 , srv1057234407:85 , srv1127273051:85 , srv1134447516:85 , srv116104166:85 , srv117255589:85 , srv1206052547:85 , srv1207662565:85 , srv1227980976:85 , srv1232893532:85 , srv1295615919:85 , srv1310585819:85 , srv1324584762:85 , srv1373104488:85 , srv1405074165:85 , srv1414193898:85 , srv1421926921:85 , srv1507114008:85 , srv1544681992:85 , srv1599826251:85 , srv1680456669:85 , srv1700683277:85 , srv1753073235:85 , srv1763139225:85 , srv1766555821:85 , srv1851531900:85 , srv1874556138:85 , srv1886165001:85 , srv1921127106:85 , srv1971939143:85 , srv2045248281:85 , srv2091906523:85 , srv2142762510:85 , srv236488901:85 , srv262356220:85 , srv26664082:85 , srv307338312:85 , srv415505413:85 , srv466630775:85 , srv478355009:85 , srv54573070:85 , srv551670353:85 , srv599252224:85 , srv65064659:85 , srv662634366:85 , srv668531860:85 , srv724085831:85 , srv743118374:85 , srv909772922:85 , srv978150082:85 , srv1004801932:86 , srv1158685569:86 , srv1385968174:86 , srv1569585196:86 , srv1594236782:86 , srv807752385:87 , srv2015613202:631 }
2023-03-03 06:54:05,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-03 06:54:05,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1131591638=9, srv668531860=86, srv54573070=74, srv2142762510=60, srv919549501=98, srv1158685569=11, srv23529383=62, srv415505413=70, srv1231575609=17, srv1683852706=39, srv1469852211=30, srv1041171784=3, srv573363651=80, srv737969943=90, srv109737493=6, srv909772922=97, srv724085831=89, srv1971939143=55, srv1134447516=10, srv31678768=68, srv1408118882=27, srv1127273051=8, srv2015613202=56, srv547115652=75, srv889739434=96, srv65064659=84, srv551670353=76, srv1700683277=40, srv1569585196=33, srv1057234407=4, srv650308076=83, srv1004801932=1, srv1232893532=18, srv558225910=77, srv335378382=69, srv117255589=13, srv307338312=67, srv1405074165=26, srv1850308486=48, srv218596602=61, srv262356220=64, srv1921127106=52, srv1004763325=0, srv1367137635=22, srv1507114008=31, srv1766555821=46, srv693635180=87, srv1874556138=50, srv978150082=99, srv1024953478=2, srv176145038=44, srv116104166=12, srv1605466460=36, srv1227980976=16, srv775357429=92, srv1098743837=7, srv1310585819=20, srv1763139225=45, srv1207662565=15, srv1385968174=25, srv743118374=91, srv2091906523=58, srv567756214=79, srv1414193898=28, srv1599826251=35, srv1967038364=54, srv478355009=72, srv1753073235=43, srv577631723=81, srv720257502=88, srv1851531900=49, srv1544681992=32, srv236488901=63, srv1421926921=29, srv561996094=78, srv1886165001=51, srv466630775=71, srv1752050456=42, srv814357057=95, srv517222357=73, srv1609527369=37, srv807752385=94, srv1071140816=5, srv1748036411=41, srv1594236782=34, srv26664082=65, srv1324584762=21, srv806531572=93, srv2045248281=57, srv1680456669=38, srv1368782436=23, srv283068948=66, srv1954097889=53, srv1843369259=47, srv2101097745=59, srv662634366=85, srv599252224=82, srv1206052547=14, srv1295615919=19, srv1373104488=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-03 06:54:05,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-03 06:54:05,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-03 06:54:05,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-03 06:54:05,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-03 06:54:05,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-03 06:54:05,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-03 06:54:05,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-03 06:54:05,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-03 06:54:05,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-03 06:54:05,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-03 06:54:05,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-03 06:54:05,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-03 06:54:05,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-03 06:54:05,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-03 06:54:05,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-03 06:54:05,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-03 06:54:05,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-03 06:54:05,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-03 06:54:05,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-03 06:54:05,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-03 06:54:05,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-03 06:54:05,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-03 06:54:05,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-03 06:54:05,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-03 06:54:05,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-03 06:54:05,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-03 06:54:05,191 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8446934863573154E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-03 06:54:05,191 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1609527369:82 , srv176145038:82 , srv889739434:82 , srv1041171784:83 , srv1469852211:83 , srv1683852706:83 , srv650308076:83 , srv1004763325:84 , srv1071140816:84 , srv109737493:84 , srv1098743837:84 , srv1131591638:84 , srv1231575609:84 , srv1367137635:84 , srv1368782436:84 , srv1408118882:84 , srv1605466460:84 , srv1748036411:84 , srv1752050456:84 , srv1843369259:84 , srv1850308486:84 , srv1954097889:84 , srv1967038364:84 , srv2101097745:84 , srv218596602:84 , srv23529383:84 , srv283068948:84 , srv31678768:84 , srv335378382:84 , srv517222357:84 , srv547115652:84 , srv558225910:84 , srv561996094:84 , srv567756214:84 , srv573363651:84 , srv577631723:84 , srv693635180:84 , srv720257502:84 , srv737969943:84 , srv775357429:84 , srv806531572:84 , srv814357057:84 , srv919549501:84 , srv1024953478:85 , srv1057234407:85 , srv1127273051:85 , srv1134447516:85 , srv116104166:85 , srv117255589:85 , srv1206052547:85 , srv1207662565:85 , srv1227980976:85 , srv1232893532:85 , srv1295615919:85 , srv1310585819:85 , srv1324584762:85 , srv1373104488:85 , srv1405074165:85 , srv1414193898:85 , srv1421926921:85 , srv1507114008:85 , srv1544681992:85 , srv1599826251:85 , srv1680456669:85 , srv1700683277:85 , srv1753073235:85 , srv1763139225:85 , srv1766555821:85 , srv1851531900:85 , srv1874556138:85 , srv1886165001:85 , srv1921127106:85 , srv1971939143:85 , srv2045248281:85 , srv2091906523:85 , srv2142762510:85 , srv236488901:85 , srv262356220:85 , srv26664082:85 , srv307338312:85 , srv415505413:85 , srv466630775:85 , srv478355009:85 , srv54573070:85 , srv551670353:85 , srv599252224:85 , srv65064659:85 , srv662634366:85 , srv668531860:85 , srv724085831:85 , srv743118374:85 , srv909772922:85 , srv978150082:85 , srv1004801932:86 , srv1158685569:86 , srv1385968174:86 , srv1569585196:86 , srv1594236782:86 , srv807752385:87 , srv2015613202:631 }
2023-03-03 06:54:05,214 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=451 (was 451), ProcessCount=171 (was 171), AvailableMemoryMB=6185 (was 6694)
2023-03-03 06:54:05,221 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=451, ProcessCount=171, AvailableMemoryMB=6184
2023-03-03 06:54:05,222 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-03 06:54:05,222 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-03 06:54:05,229 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1093490560:84 , srv1187298050:84 , srv1386228662:84 , srv1431248878:84 , srv1511184388:84 , srv1530151239:84 , srv1556970287:84 , srv157715289:84 , srv170814611:84 , srv1715730984:84 , srv1752540581:84 , srv176810161:84 , srv1773184852:84 , srv1777526474:84 , srv180545036:84 , srv1923521610:84 , srv2089127515:84 , srv2115739081:84 , srv338224714:84 , srv384295194:84 , srv517807777:84 , srv535752853:84 , srv588038360:84 , srv71427782:84 , srv757819412:84 , srv843947539:84 , srv852456763:84 , srv962664014:84 , srv980391008:84 , srv1368274005:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-03 06:54:05,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-03 06:54:05,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1368274005=2, srv1386228662=3, srv180545036=15, srv1093490560=0, srv1752540581=11, srv1715730984=10, srv588038360=23, srv980391008=29, srv176810161=12, srv1923521610=16, srv535752853=22, srv2089127515=17, srv757819412=25, srv962664014=28, srv1431248878=4, srv2115739081=18, srv1530151239=6, srv1187298050=1, srv338224714=19, srv384295194=20, srv157715289=8, srv1773184852=13, srv71427782=24, srv1511184388=5, srv170814611=9, srv852456763=27, srv843947539=26, srv1777526474=14, srv1556970287=7, srv517807777=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-03 06:54:05,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-03 06:54:05,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-03 06:54:05,257 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.071356056837792, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-03 06:54:05,258 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912417883869387, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.071356056837792, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-03 06:56:05,242 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6707320 different iterations.  Found a solution that moves 1844 regions; Going from a computed imbalance of 0.9912417883869387 to a new imbalance of 4.303987990979898E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.682962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-03 06:56:05,249 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1093490560:90 , srv1187298050:90 , srv1368274005:90 , srv1386228662:90 , srv1431248878:90 , srv1511184388:90 , srv1530151239:90 , srv1556970287:90 , srv157715289:90 , srv170814611:90 , srv1715730984:90 , srv1752540581:90 , srv176810161:90 , srv1773184852:90 , srv1777526474:90 , srv180545036:90 , srv1923521610:90 , srv2089127515:90 , srv2115739081:90 , srv338224714:90 , srv384295194:90 , srv517807777:90 , srv535752853:90 , srv588038360:90 , srv71427782:90 , srv757819412:90 , srv843947539:90 , srv852456763:90 , srv962664014:90 , srv980391008:90 }
2023-03-03 06:56:05,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-03 06:56:05,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1368274005=2, srv1386228662=3, srv180545036=15, srv1093490560=0, srv1752540581=11, srv1715730984=10, srv588038360=23, srv980391008=29, srv176810161=12, srv1923521610=16, srv535752853=22, srv2089127515=17, srv757819412=25, srv962664014=28, srv1431248878=4, srv2115739081=18, srv1530151239=6, srv1187298050=1, srv338224714=19, srv384295194=20, srv157715289=8, srv1773184852=13, srv71427782=24, srv1511184388=5, srv170814611=9, srv852456763=27, srv843947539=26, srv1777526474=14, srv1556970287=7, srv517807777=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-03 06:56:05,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-03 06:56:05,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-03 06:56:05,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-03 06:56:05,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-03 06:56:05,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-03 06:56:05,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-03 06:56:05,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-03 06:56:05,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-03 06:56:05,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-03 06:56:05,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-03 06:56:05,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-03 06:56:05,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-03 06:56:05,273 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-03 06:56:05,273 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1093490560:90 , srv1187298050:90 , srv1368274005:90 , srv1386228662:90 , srv1431248878:90 , srv1511184388:90 , srv1530151239:90 , srv1556970287:90 , srv157715289:90 , srv170814611:90 , srv1715730984:90 , srv1752540581:90 , srv176810161:90 , srv1773184852:90 , srv1777526474:90 , srv180545036:90 , srv1923521610:90 , srv2089127515:90 , srv2115739081:90 , srv338224714:90 , srv384295194:90 , srv517807777:90 , srv535752853:90 , srv588038360:90 , srv71427782:90 , srv757819412:90 , srv843947539:90 , srv852456763:90 , srv962664014:90 , srv980391008:90 }
2023-03-03 06:56:05,291 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=183 (was 451), ProcessCount=170 (was 171), AvailableMemoryMB=5971 (was 6184)