2023-04-19 07:55:42,526 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-04-19 07:55:42,910 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-04-19 07:55:43,055 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-04-19 07:55:43,131 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-19 07:55:43,139 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-19 07:55:43,156 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=493, ProcessCount=169, AvailableMemoryMB=7081
2023-04-19 07:55:43,159 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-19 07:55:43,160 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-19 07:55:43,322 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1065119428:84 , srv108254748:84 , srv1082598124:84 , srv1088940804:84 , srv110438189:84 , srv1122350296:84 , srv1126514153:84 , srv112893551:84 , srv1134683512:84 , srv1136423920:84 , srv1138953951:84 , srv1144970331:84 , srv1199375620:84 , srv1199579335:84 , srv1270732358:84 , srv1278502201:84 , srv1325327198:84 , srv132550387:84 , srv1366974649:84 , srv1368608234:84 , srv1393471410:84 , srv1397049026:84 , srv14182831:84 , srv1449461933:84 , srv1454220165:84 , srv1468319349:84 , srv1501251422:84 , srv1508322725:84 , srv1543520625:84 , srv1628117962:84 , srv1657435375:84 , srv1660128355:84 , srv1670231935:84 , srv1712990979:84 , srv171902737:84 , srv175964706:84 , srv1766108538:84 , srv1779097523:84 , srv1779143975:84 , srv1804228706:84 , srv180502434:84 , srv1881579877:84 , srv1882801185:84 , srv1890374930:84 , srv1932027700:84 , srv1972444370:84 , srv1986135930:84 , srv1999220047:84 , srv2009116930:84 , srv2058078880:84 , srv2080976329:84 , srv209300775:84 , srv2097308309:84 , srv2097512782:84 , srv212335045:84 , srv2131622724:84 , srv2144780204:84 , srv219422331:84 , srv23023177:84 , srv234662307:84 , srv344570142:84 , srv356548732:84 , srv379460509:84 , srv439418552:84 , srv445375883:84 , srv448958087:84 , srv467275525:84 , srv474698480:84 , srv477897885:84 , srv521876955:84 , srv530941609:84 , srv542846646:84 , srv551208770:84 , srv555475836:84 , srv559022863:84 , srv563602305:84 , srv566319847:84 , srv569110689:84 , srv579805592:84 , srv635115665:84 , srv729273045:84 , srv762591687:84 , srv766673255:84 , srv786956634:84 , srv787008750:84 , srv812115453:84 , srv81539076:84 , srv842843135:84 , srv85280590:84 , srv870080345:84 , srv88469063:84 , srv895070824:84 , srv902846380:84 , srv909833325:84 , srv925070229:84 , srv953439969:84 , srv968677871:84 , srv975033004:84 , srv993480116:84 , srv26790588:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-04-19 07:55:43,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-19 07:55:43,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1882801185=42, srv1986135930=46, srv14182831=22, srv467275525=67, srv1449461933=23, srv1779097523=37, srv559022863=75, srv356548732=62, srv112893551=7, srv1660128355=31, srv870080345=90, srv925070229=95, srv477897885=69, srv1397049026=21, srv1122350296=5, srv1468319349=25, srv1932027700=44, srv909833325=94, srv175964706=35, srv787008750=85, srv1325327198=16, srv2080976329=50, srv762591687=82, srv110438189=4, srv563602305=76, srv85280590=89, srv1881579877=41, srv234662307=59, srv1366974649=18, srv1712990979=33, srv1501251422=26, srv1082598124=2, srv1126514153=6, srv895070824=92, srv2009116930=48, srv344570142=61, srv1766108538=36, srv1065119428=0, srv439418552=64, srv521876955=70, srv379460509=63, srv180502434=40, srv1670231935=32, srv902846380=93, srv729273045=81, srv1199375620=12, srv1278502201=15, srv1804228706=39, srv26790588=60, srv551208770=73, srv1779143975=38, srv209300775=51, srv448958087=66, srv171902737=34, srv1999220047=47, srv993480116=99, srv2144780204=56, srv566319847=77, srv1138953951=10, srv1393471410=20, srv1136423920=9, srv786956634=84, srv1199579335=13, srv1508322725=27, srv1454220165=24, srv569110689=78, srv474698480=68, srv635115665=80, srv1972444370=45, srv579805592=79, srv23023177=58, srv530941609=71, srv81539076=87, srv108254748=1, srv975033004=98, srv542846646=72, srv1134683512=8, srv1628117962=29, srv2058078880=49, srv132550387=17, srv555475836=74, srv2097512782=53, srv445375883=65, srv2097308309=52, srv1657435375=30, srv1368608234=19, srv212335045=54, srv1088940804=3, srv1890374930=43, srv766673255=83, srv88469063=91, srv2131622724=55, srv1270732358=14, srv842843135=88, srv219422331=57, srv812115453=86, srv1144970331=11, srv1543520625=28, srv968677871=97, srv953439969=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-19 07:55:43,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-19 07:55:43,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-19 07:55:43,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-19 07:55:43,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-19 07:55:43,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-19 07:55:43,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-19 07:55:43,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-19 07:55:43,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-19 07:55:43,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-19 07:55:43,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-19 07:55:43,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-19 07:55:43,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-19 07:55:43,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-19 07:55:43,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-19 07:55:43,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-19 07:55:43,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-19 07:55:43,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-19 07:55:43,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-19 07:55:43,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-19 07:55:43,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-19 07:55:43,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-19 07:55:43,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-19 07:55:43,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-19 07:55:43,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-19 07:55:43,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-19 07:55:43,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-19 07:55:43,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-19 07:55:43,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-19 07:55:43,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-19 07:55:43,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-19 07:55:43,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-19 07:55:43,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-19 07:55:43,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-19 07:55:43,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-19 07:55:43,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-19 07:55:43,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-19 07:55:43,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-19 07:55:43,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-19 07:55:43,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-19 07:55:43,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-19 07:55:43,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-19 07:55:43,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-19 07:55:43,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-19 07:55:43,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-19 07:55:43,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-19 07:55:43,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-19 07:55:43,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-19 07:55:43,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-19 07:55:43,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-19 07:55:43,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-19 07:55:43,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-19 07:55:43,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-19 07:55:43,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-19 07:55:43,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-19 07:55:43,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-19 07:55:43,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-19 07:55:43,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-19 07:55:43,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-19 07:55:43,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-19 07:55:43,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-19 07:55:43,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-19 07:55:43,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-19 07:55:43,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-19 07:55:43,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-19 07:55:43,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-19 07:55:43,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-19 07:55:43,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-19 07:55:43,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-19 07:55:43,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-19 07:55:43,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-19 07:55:43,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-19 07:55:43,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-19 07:55:43,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-19 07:55:43,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-19 07:55:43,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-19 07:55:43,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-19 07:55:43,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-19 07:55:43,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-19 07:55:43,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-19 07:55:43,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-19 07:55:43,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-19 07:55:43,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-19 07:55:43,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-19 07:55:43,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-19 07:55:43,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-19 07:55:43,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-19 07:55:43,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-19 07:55:43,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-19 07:55:43,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-19 07:55:43,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-19 07:55:43,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-19 07:55:43,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-19 07:55:43,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-19 07:55:43,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-19 07:55:43,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-19 07:55:43,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-19 07:55:43,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-19 07:55:43,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-19 07:55:43,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-19 07:55:43,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-19 07:55:43,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-19 07:55:43,735 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-19 07:55:43,736 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-19 07:55:43,736 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-19 07:55:44,195 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 685 ms to try 5000 different iterations.  Found a solution that moves 4694 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1796819710851388. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0388312865042161, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0565506834439673, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5215555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0388312865042161, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17491666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-19 07:55:44,212 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv108254748:82 , srv953439969:82 , srv993480116:82 , srv1065119428:83 , srv132550387:83 , srv1766108538:83 , srv1882801185:83 , srv1999220047:83 , srv209300775:83 , srv2131622724:83 , srv1366974649:84 , srv1368608234:84 , srv1397049026:84 , srv1501251422:84 , srv1779097523:84 , srv2097308309:84 , srv219422331:84 , srv448958087:84 , srv559022863:84 , srv563602305:84 , srv729273045:84 , srv762591687:84 , srv81539076:84 , srv895070824:84 , srv1088940804:85 , srv112893551:85 , srv1136423920:85 , srv1278502201:85 , srv14182831:85 , srv1449461933:85 , srv344570142:85 , srv356548732:85 , srv477897885:85 , srv569110689:85 , srv766673255:85 , srv870080345:85 , srv925070229:85 , srv1138953951:86 , srv1468319349:86 , srv1543520625:86 , srv1657435375:86 , srv171902737:86 , srv1779143975:86 , srv234662307:86 , srv379460509:86 , srv521876955:86 , srv542846646:86 , srv551208770:86 , srv555475836:86 , srv902846380:86 , srv975033004:86 , srv1082598124:87 , srv1270732358:87 , srv1660128355:87 , srv1712990979:87 , srv1881579877:87 , srv1972444370:87 , srv2058078880:87 , srv23023177:87 , srv439418552:87 , srv474698480:87 , srv530941609:87 , srv579805592:87 , srv635115665:87 , srv812115453:87 , srv842843135:87 , srv85280590:87 , srv88469063:87 , srv909833325:87 , srv968677871:87 , srv110438189:88 , srv1122350296:88 , srv1325327198:88 , srv1393471410:88 , srv175964706:88 , srv180502434:88 , srv2009116930:88 , srv212335045:88 , srv2144780204:88 , srv467275525:88 , srv786956634:88 , srv1508322725:89 , srv1628117962:89 , srv1804228706:89 , srv1890374930:89 , srv2080976329:89 , srv445375883:89 , srv566319847:89 , srv1126514153:90 , srv1986135930:90 , srv2097512782:90 , srv1144970331:91 , srv1199375620:91 , srv1199579335:91 , srv1454220165:91 , srv1670231935:91 , srv1134683512:92 , srv1932027700:96 , srv787008750:96 , srv26790588:435 }
2023-04-19 07:55:44,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-19 07:55:44,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1882801185=42, srv1986135930=46, srv14182831=22, srv467275525=67, srv1449461933=23, srv1779097523=37, srv559022863=75, srv356548732=62, srv112893551=7, srv1660128355=31, srv870080345=90, srv925070229=95, srv477897885=69, srv1397049026=21, srv1122350296=5, srv1468319349=25, srv1932027700=44, srv909833325=94, srv175964706=35, srv787008750=85, srv1325327198=16, srv2080976329=50, srv762591687=82, srv110438189=4, srv563602305=76, srv85280590=89, srv1881579877=41, srv234662307=59, srv1366974649=18, srv1712990979=33, srv1501251422=26, srv1082598124=2, srv1126514153=6, srv895070824=92, srv2009116930=48, srv344570142=61, srv1766108538=36, srv1065119428=0, srv439418552=64, srv521876955=70, srv379460509=63, srv180502434=40, srv1670231935=32, srv902846380=93, srv729273045=81, srv1199375620=12, srv1278502201=15, srv1804228706=39, srv26790588=60, srv551208770=73, srv1779143975=38, srv209300775=51, srv448958087=66, srv171902737=34, srv1999220047=47, srv993480116=99, srv2144780204=56, srv566319847=77, srv1138953951=10, srv1393471410=20, srv1136423920=9, srv786956634=84, srv1199579335=13, srv1508322725=27, srv1454220165=24, srv569110689=78, srv474698480=68, srv635115665=80, srv1972444370=45, srv579805592=79, srv23023177=58, srv530941609=71, srv81539076=87, srv108254748=1, srv975033004=98, srv542846646=72, srv1134683512=8, srv1628117962=29, srv2058078880=49, srv132550387=17, srv555475836=74, srv2097512782=53, srv445375883=65, srv2097308309=52, srv1657435375=30, srv1368608234=19, srv212335045=54, srv1088940804=3, srv1890374930=43, srv766673255=83, srv88469063=91, srv2131622724=55, srv1270732358=14, srv842843135=88, srv219422331=57, srv812115453=86, srv1144970331=11, srv1543520625=28, srv968677871=97, srv953439969=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-19 07:55:44,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-19 07:55:44,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-19 07:55:44,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-19 07:55:44,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-19 07:55:44,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-19 07:55:44,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-19 07:55:44,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-19 07:55:44,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-19 07:55:44,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-19 07:55:44,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-19 07:55:44,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-19 07:55:44,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-19 07:55:44,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-19 07:55:44,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-19 07:55:44,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-19 07:55:44,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-19 07:55:44,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-19 07:55:44,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-19 07:55:44,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-19 07:55:44,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-19 07:55:44,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-19 07:55:44,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-19 07:55:44,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-19 07:55:44,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-19 07:55:44,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-19 07:55:44,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-19 07:55:44,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-19 07:55:44,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-19 07:55:44,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-19 07:55:44,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-19 07:55:44,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-19 07:55:44,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-19 07:55:44,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-19 07:55:44,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-19 07:55:44,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-19 07:55:44,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-19 07:55:44,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-19 07:55:44,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-19 07:55:44,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-19 07:55:44,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-19 07:55:44,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-19 07:55:44,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-19 07:55:44,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-19 07:55:44,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-19 07:55:44,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-19 07:55:44,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-19 07:55:44,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-19 07:55:44,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-19 07:55:44,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-19 07:55:44,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-19 07:55:44,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-19 07:55:44,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-19 07:55:44,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-19 07:55:44,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-19 07:55:44,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-19 07:55:44,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-19 07:55:44,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-19 07:55:44,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-19 07:55:44,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-19 07:55:44,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-19 07:55:44,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-19 07:55:44,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-19 07:55:44,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-19 07:55:44,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-19 07:55:44,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-19 07:55:44,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-19 07:55:44,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-19 07:55:44,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-19 07:55:44,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-19 07:55:44,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-19 07:55:44,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-19 07:55:44,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-19 07:55:44,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-19 07:55:44,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-19 07:55:44,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-19 07:55:44,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-19 07:55:44,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-19 07:55:44,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-19 07:55:44,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-19 07:55:44,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-19 07:55:44,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-19 07:55:44,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-19 07:55:44,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-19 07:55:44,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-19 07:55:44,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-19 07:55:44,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-19 07:55:44,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-19 07:55:44,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-19 07:55:44,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-19 07:55:44,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-19 07:55:44,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-19 07:55:44,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-19 07:55:44,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-19 07:55:44,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-19 07:55:44,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-19 07:55:44,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-19 07:55:44,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-19 07:55:44,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-19 07:55:44,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-19 07:55:44,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-19 07:55:44,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-19 07:55:44,430 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0388312865042161, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0565506834439673, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0388312865042161, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17491666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-19 07:55:44,430 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-19 07:55:44,430 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17964910299463505, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0388312865042161, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0565506834439673, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0388312865042161, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17491666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-19 07:55:44,735 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 438 ms to try 5000 different iterations.  Found a solution that moves 2667 regions; Going from a computed imbalance of 0.17964910299463505 to a new imbalance of 0.015551519763017779. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0212837761469435); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03624334762288909, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29633333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0212837761469435); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-19 07:55:44,745 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1932027700:74 , srv1134683512:80 , srv1670231935:81 , srv2097512782:81 , srv445375883:81 , srv1122350296:82 , srv1144970331:82 , srv1199375620:82 , srv1454220165:82 , srv1890374930:82 , srv1986135930:82 , srv786956634:82 , srv787008750:82 , srv1126514153:83 , srv1199579335:83 , srv1325327198:83 , srv175964706:83 , srv2144780204:83 , srv530941609:83 , srv551208770:83 , srv566319847:83 , srv975033004:83 , srv110438189:84 , srv1270732358:84 , srv1508322725:84 , srv1660128355:84 , srv2009116930:84 , srv2080976329:84 , srv212335045:84 , srv1082598124:85 , srv1393471410:85 , srv1628117962:85 , srv171902737:85 , srv1804228706:85 , srv1972444370:85 , srv439418552:85 , srv474698480:85 , srv477897885:85 , srv555475836:85 , srv635115665:85 , srv766673255:85 , srv842843135:85 , srv88469063:85 , srv968677871:85 , srv112893551:86 , srv1136423920:86 , srv1278502201:86 , srv1468319349:86 , srv1766108538:86 , srv180502434:86 , srv2058078880:86 , srv23023177:86 , srv448958087:86 , srv467275525:86 , srv542846646:86 , srv579805592:86 , srv85280590:86 , srv870080345:86 , srv1138953951:87 , srv1366974649:87 , srv1368608234:87 , srv1397049026:87 , srv14182831:87 , srv1449461933:87 , srv1543520625:87 , srv1657435375:87 , srv1881579877:87 , srv2097308309:87 , srv219422331:87 , srv234662307:87 , srv344570142:87 , srv563602305:87 , srv569110689:87 , srv729273045:87 , srv812115453:87 , srv81539076:87 , srv895070824:87 , srv902846380:87 , srv1088940804:88 , srv1882801185:88 , srv209300775:88 , srv2131622724:88 , srv379460509:88 , srv559022863:88 , srv762591687:88 , srv909833325:88 , srv925070229:88 , srv1065119428:89 , srv132550387:89 , srv1779097523:89 , srv1779143975:89 , srv1999220047:89 , srv356548732:89 , srv953439969:89 , srv993480116:89 , srv108254748:90 , srv1712990979:90 , srv521876955:90 , srv1501251422:91 , srv26790588:528 }
2023-04-19 07:55:44,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-19 07:55:44,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1882801185=42, srv1986135930=46, srv14182831=22, srv467275525=67, srv1449461933=23, srv1779097523=37, srv559022863=75, srv356548732=62, srv112893551=7, srv1660128355=31, srv870080345=90, srv925070229=95, srv477897885=69, srv1397049026=21, srv1122350296=5, srv1468319349=25, srv1932027700=44, srv909833325=94, srv175964706=35, srv787008750=85, srv1325327198=16, srv2080976329=50, srv762591687=82, srv110438189=4, srv563602305=76, srv85280590=89, srv1881579877=41, srv234662307=59, srv1366974649=18, srv1712990979=33, srv1501251422=26, srv1082598124=2, srv1126514153=6, srv895070824=92, srv2009116930=48, srv344570142=61, srv1766108538=36, srv1065119428=0, srv439418552=64, srv521876955=70, srv379460509=63, srv180502434=40, srv1670231935=32, srv902846380=93, srv729273045=81, srv1199375620=12, srv1278502201=15, srv1804228706=39, srv26790588=60, srv551208770=73, srv1779143975=38, srv209300775=51, srv448958087=66, srv171902737=34, srv1999220047=47, srv993480116=99, srv2144780204=56, srv566319847=77, srv1138953951=10, srv1393471410=20, srv1136423920=9, srv786956634=84, srv1199579335=13, srv1508322725=27, srv1454220165=24, srv569110689=78, srv474698480=68, srv635115665=80, srv1972444370=45, srv579805592=79, srv23023177=58, srv530941609=71, srv81539076=87, srv108254748=1, srv975033004=98, srv542846646=72, srv1134683512=8, srv1628117962=29, srv2058078880=49, srv132550387=17, srv555475836=74, srv2097512782=53, srv445375883=65, srv2097308309=52, srv1657435375=30, srv1368608234=19, srv212335045=54, srv1088940804=3, srv1890374930=43, srv766673255=83, srv88469063=91, srv2131622724=55, srv1270732358=14, srv842843135=88, srv219422331=57, srv812115453=86, srv1144970331=11, srv1543520625=28, srv968677871=97, srv953439969=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-19 07:55:44,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-19 07:55:44,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-19 07:55:44,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-19 07:55:44,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-19 07:55:44,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-19 07:55:44,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-19 07:55:44,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-19 07:55:44,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-19 07:55:44,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-19 07:55:44,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-19 07:55:44,886 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0212837761469435); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03624334762288909, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0212837761469435); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-19 07:55:44,886 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-19 07:55:44,886 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015532845029874705, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0212837761469435); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03624334762288909, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0212837761469435); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-19 07:55:45,203 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 414 ms to try 5000 different iterations.  Found a solution that moves 936 regions; Going from a computed imbalance of 0.015532845029874705 to a new imbalance of 4.602881907199587E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00880572771006482); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01471006740476568); MoveCostFunction : (multiplier=7.0, imbalance=0.104, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00880572771006482); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-19 07:55:45,207 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1712990979:79 , srv787008750:80 , srv1779143975:81 , srv521876955:81 , srv1501251422:82 , srv1126514153:83 , srv1138953951:83 , srv1199579335:83 , srv1628117962:83 , srv1804228706:83 , srv180502434:83 , srv1881579877:83 , srv23023177:83 , srv356548732:83 , srv902846380:83 , srv909833325:83 , srv1065119428:84 , srv1199375620:84 , srv1454220165:84 , srv1543520625:84 , srv1779097523:84 , srv2009116930:84 , srv2058078880:84 , srv2080976329:84 , srv344570142:84 , srv379460509:84 , srv467275525:84 , srv579805592:84 , srv812115453:84 , srv85280590:84 , srv925070229:84 , srv108254748:85 , srv1082598124:85 , srv1088940804:85 , srv1134683512:85 , srv1144970331:85 , srv1270732358:85 , srv132550387:85 , srv1393471410:85 , srv1449461933:85 , srv1468319349:85 , srv1660128355:85 , srv171902737:85 , srv1972444370:85 , srv1986135930:85 , srv1999220047:85 , srv2097308309:85 , srv212335045:85 , srv234662307:85 , srv439418552:85 , srv542846646:85 , srv635115665:85 , srv110438189:86 , srv112893551:86 , srv1278502201:86 , srv1325327198:86 , srv1366974649:86 , srv1368608234:86 , srv1397049026:86 , srv1508322725:86 , srv1657435375:86 , srv1670231935:86 , srv175964706:86 , srv1882801185:86 , srv2097512782:86 , srv2144780204:86 , srv219422331:86 , srv445375883:86 , srv448958087:86 , srv474698480:86 , srv477897885:86 , srv555475836:86 , srv559022863:86 , srv566319847:86 , srv569110689:86 , srv762591687:86 , srv786956634:86 , srv81539076:86 , srv842843135:86 , srv870080345:86 , srv895070824:86 , srv953439969:86 , srv1122350296:87 , srv1136423920:87 , srv14182831:87 , srv1766108538:87 , srv1932027700:87 , srv209300775:87 , srv530941609:87 , srv563602305:87 , srv729273045:87 , srv968677871:87 , srv993480116:87 , srv1890374930:88 , srv2131622724:88 , srv551208770:88 , srv88469063:88 , srv975033004:88 , srv766673255:89 , srv26790588:573 }
2023-04-19 07:55:45,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-19 07:55:45,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1882801185=42, srv1986135930=46, srv14182831=22, srv467275525=67, srv1449461933=23, srv1779097523=37, srv559022863=75, srv356548732=62, srv112893551=7, srv1660128355=31, srv870080345=90, srv925070229=95, srv477897885=69, srv1397049026=21, srv1122350296=5, srv1468319349=25, srv1932027700=44, srv909833325=94, srv175964706=35, srv787008750=85, srv1325327198=16, srv2080976329=50, srv762591687=82, srv110438189=4, srv563602305=76, srv85280590=89, srv1881579877=41, srv234662307=59, srv1366974649=18, srv1712990979=33, srv1501251422=26, srv1082598124=2, srv1126514153=6, srv895070824=92, srv2009116930=48, srv344570142=61, srv1766108538=36, srv1065119428=0, srv439418552=64, srv521876955=70, srv379460509=63, srv180502434=40, srv1670231935=32, srv902846380=93, srv729273045=81, srv1199375620=12, srv1278502201=15, srv1804228706=39, srv26790588=60, srv551208770=73, srv1779143975=38, srv209300775=51, srv448958087=66, srv171902737=34, srv1999220047=47, srv993480116=99, srv2144780204=56, srv566319847=77, srv1138953951=10, srv1393471410=20, srv1136423920=9, srv786956634=84, srv1199579335=13, srv1508322725=27, srv1454220165=24, srv569110689=78, srv474698480=68, srv635115665=80, srv1972444370=45, srv579805592=79, srv23023177=58, srv530941609=71, srv81539076=87, srv108254748=1, srv975033004=98, srv542846646=72, srv1134683512=8, srv1628117962=29, srv2058078880=49, srv132550387=17, srv555475836=74, srv2097512782=53, srv445375883=65, srv2097308309=52, srv1657435375=30, srv1368608234=19, srv212335045=54, srv1088940804=3, srv1890374930=43, srv766673255=83, srv88469063=91, srv2131622724=55, srv1270732358=14, srv842843135=88, srv219422331=57, srv812115453=86, srv1144970331=11, srv1543520625=28, srv968677871=97, srv953439969=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-19 07:55:45,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-19 07:55:45,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-19 07:55:45,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-19 07:55:45,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-19 07:55:45,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-19 07:55:45,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-19 07:55:45,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-19 07:55:45,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-19 07:55:45,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-19 07:55:45,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-19 07:55:45,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-19 07:55:45,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-19 07:55:45,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-19 07:55:45,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-19 07:55:45,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-19 07:55:45,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-19 07:55:45,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-19 07:55:45,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-19 07:55:45,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-19 07:55:45,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-19 07:55:45,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-19 07:55:45,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-19 07:55:45,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-19 07:55:45,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-19 07:55:45,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-19 07:55:45,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-19 07:55:45,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-19 07:55:45,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-19 07:55:45,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-19 07:55:45,356 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00880572771006482); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01471006740476568); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00880572771006482); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-19 07:55:45,356 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-19 07:55:45,356 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.537341786382496E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00880572771006482); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01471006740476568); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00880572771006482); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-19 07:55:45,631 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 374 ms to try 5000 different iterations.  Found a solution that moves 279 regions; Going from a computed imbalance of 4.537341786382496E-4 to a new imbalance of 1.2185111100820343E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.031, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-19 07:55:45,632 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv88469063:82 , srv110438189:83 , srv14182831:83 , srv1508322725:83 , srv1657435375:83 , srv1890374930:83 , srv2131622724:83 , srv766673255:83 , srv968677871:83 , srv1136423920:84 , srv1144970331:84 , srv1393471410:84 , srv1543520625:84 , srv1670231935:84 , srv209300775:84 , srv234662307:84 , srv379460509:84 , srv467275525:84 , srv474698480:84 , srv542846646:84 , srv555475836:84 , srv559022863:84 , srv563602305:84 , srv569110689:84 , srv729273045:84 , srv762591687:84 , srv787008750:84 , srv812115453:84 , srv842843135:84 , srv895070824:84 , srv909833325:84 , srv993480116:84 , srv108254748:85 , srv1082598124:85 , srv1088940804:85 , srv1122350296:85 , srv1134683512:85 , srv1199375620:85 , srv1199579335:85 , srv1278502201:85 , srv1325327198:85 , srv1366974649:85 , srv1368608234:85 , srv1397049026:85 , srv1449461933:85 , srv1454220165:85 , srv1468319349:85 , srv1501251422:85 , srv1628117962:85 , srv175964706:85 , srv1779097523:85 , srv1804228706:85 , srv180502434:85 , srv1881579877:85 , srv1882801185:85 , srv1932027700:85 , srv1972444370:85 , srv1986135930:85 , srv1999220047:85 , srv2058078880:85 , srv2080976329:85 , srv2097512782:85 , srv212335045:85 , srv2144780204:85 , srv219422331:85 , srv356548732:85 , srv439418552:85 , srv521876955:85 , srv530941609:85 , srv551208770:85 , srv566319847:85 , srv579805592:85 , srv635115665:85 , srv81539076:85 , srv85280590:85 , srv870080345:85 , srv925070229:85 , srv953439969:85 , srv975033004:85 , srv1065119428:86 , srv1126514153:86 , srv112893551:86 , srv1138953951:86 , srv1270732358:86 , srv132550387:86 , srv1660128355:86 , srv171902737:86 , srv1766108538:86 , srv1779143975:86 , srv2009116930:86 , srv2097308309:86 , srv23023177:86 , srv344570142:86 , srv445375883:86 , srv448958087:86 , srv477897885:86 , srv786956634:86 , srv902846380:86 , srv1712990979:87 , srv26790588:606 }
2023-04-19 07:55:45,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-19 07:55:45,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1882801185=42, srv1986135930=46, srv14182831=22, srv467275525=67, srv1449461933=23, srv1779097523=37, srv559022863=75, srv356548732=62, srv112893551=7, srv1660128355=31, srv870080345=90, srv925070229=95, srv477897885=69, srv1397049026=21, srv1122350296=5, srv1468319349=25, srv1932027700=44, srv909833325=94, srv175964706=35, srv787008750=85, srv1325327198=16, srv2080976329=50, srv762591687=82, srv110438189=4, srv563602305=76, srv85280590=89, srv1881579877=41, srv234662307=59, srv1366974649=18, srv1712990979=33, srv1501251422=26, srv1082598124=2, srv1126514153=6, srv895070824=92, srv2009116930=48, srv344570142=61, srv1766108538=36, srv1065119428=0, srv439418552=64, srv521876955=70, srv379460509=63, srv180502434=40, srv1670231935=32, srv902846380=93, srv729273045=81, srv1199375620=12, srv1278502201=15, srv1804228706=39, srv26790588=60, srv551208770=73, srv1779143975=38, srv209300775=51, srv448958087=66, srv171902737=34, srv1999220047=47, srv993480116=99, srv2144780204=56, srv566319847=77, srv1138953951=10, srv1393471410=20, srv1136423920=9, srv786956634=84, srv1199579335=13, srv1508322725=27, srv1454220165=24, srv569110689=78, srv474698480=68, srv635115665=80, srv1972444370=45, srv579805592=79, srv23023177=58, srv530941609=71, srv81539076=87, srv108254748=1, srv975033004=98, srv542846646=72, srv1134683512=8, srv1628117962=29, srv2058078880=49, srv132550387=17, srv555475836=74, srv2097512782=53, srv445375883=65, srv2097308309=52, srv1657435375=30, srv1368608234=19, srv212335045=54, srv1088940804=3, srv1890374930=43, srv766673255=83, srv88469063=91, srv2131622724=55, srv1270732358=14, srv842843135=88, srv219422331=57, srv812115453=86, srv1144970331=11, srv1543520625=28, srv968677871=97, srv953439969=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-19 07:55:45,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-19 07:55:45,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-19 07:55:45,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-19 07:55:45,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-19 07:55:45,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-19 07:55:45,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-19 07:55:45,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-19 07:55:45,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-19 07:55:45,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-19 07:55:45,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-19 07:55:45,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-19 07:55:45,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-19 07:55:45,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-19 07:55:45,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-19 07:55:45,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-19 07:55:45,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-19 07:55:45,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-19 07:55:45,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-19 07:55:45,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-19 07:55:45,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-19 07:55:45,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-19 07:55:45,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-19 07:55:45,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-19 07:55:45,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-19 07:55:45,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-19 07:55:45,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-19 07:55:45,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-19 07:55:45,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-19 07:55:45,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-19 07:55:45,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-19 07:55:45,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-19 07:55:45,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-19 07:55:45,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-19 07:55:45,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-19 07:55:45,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-19 07:55:45,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-19 07:55:45,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-19 07:55:45,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-19 07:55:45,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-19 07:55:45,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-19 07:55:45,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-19 07:55:45,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-19 07:55:45,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-19 07:55:45,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-19 07:55:45,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-19 07:55:45,768 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.0231511345695518E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-19 07:55:45,768 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv88469063:82 , srv110438189:83 , srv14182831:83 , srv1508322725:83 , srv1657435375:83 , srv1890374930:83 , srv2131622724:83 , srv766673255:83 , srv968677871:83 , srv1136423920:84 , srv1144970331:84 , srv1393471410:84 , srv1543520625:84 , srv1670231935:84 , srv209300775:84 , srv234662307:84 , srv379460509:84 , srv467275525:84 , srv474698480:84 , srv542846646:84 , srv555475836:84 , srv559022863:84 , srv563602305:84 , srv569110689:84 , srv729273045:84 , srv762591687:84 , srv787008750:84 , srv812115453:84 , srv842843135:84 , srv895070824:84 , srv909833325:84 , srv993480116:84 , srv108254748:85 , srv1082598124:85 , srv1088940804:85 , srv1122350296:85 , srv1134683512:85 , srv1199375620:85 , srv1199579335:85 , srv1278502201:85 , srv1325327198:85 , srv1366974649:85 , srv1368608234:85 , srv1397049026:85 , srv1449461933:85 , srv1454220165:85 , srv1468319349:85 , srv1501251422:85 , srv1628117962:85 , srv175964706:85 , srv1779097523:85 , srv1804228706:85 , srv180502434:85 , srv1881579877:85 , srv1882801185:85 , srv1932027700:85 , srv1972444370:85 , srv1986135930:85 , srv1999220047:85 , srv2058078880:85 , srv2080976329:85 , srv2097512782:85 , srv212335045:85 , srv2144780204:85 , srv219422331:85 , srv356548732:85 , srv439418552:85 , srv521876955:85 , srv530941609:85 , srv551208770:85 , srv566319847:85 , srv579805592:85 , srv635115665:85 , srv81539076:85 , srv85280590:85 , srv870080345:85 , srv925070229:85 , srv953439969:85 , srv975033004:85 , srv1065119428:86 , srv1126514153:86 , srv112893551:86 , srv1138953951:86 , srv1270732358:86 , srv132550387:86 , srv1660128355:86 , srv171902737:86 , srv1766108538:86 , srv1779143975:86 , srv2009116930:86 , srv2097308309:86 , srv23023177:86 , srv344570142:86 , srv445375883:86 , srv448958087:86 , srv477897885:86 , srv786956634:86 , srv902846380:86 , srv1712990979:87 , srv26790588:606 }
2023-04-19 07:55:45,803 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=493 (was 493), ProcessCount=169 (was 169), AvailableMemoryMB=6598 (was 7081)
2023-04-19 07:55:45,811 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=493, ProcessCount=169, AvailableMemoryMB=6598
2023-04-19 07:55:45,812 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-19 07:55:45,813 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-04-19 07:55:45,828 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1034407790:84 , srv1076637379:84 , srv1168967785:84 , srv1174510867:84 , srv1220879565:84 , srv1225432286:84 , srv131989639:84 , srv1370108999:84 , srv157907148:84 , srv1739704627:84 , srv1779836429:84 , srv1833968024:84 , srv1859297487:84 , srv1880297232:84 , srv2062928799:84 , srv213031447:84 , srv2533511:84 , srv302754015:84 , srv429942958:84 , srv43285656:84 , srv43335643:84 , srv550313867:84 , srv623409017:84 , srv728512903:84 , srv796544135:84 , srv827831503:84 , srv939392979:84 , srv941264076:84 , srv98107085:84 , srv1864592603:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-04-19 07:55:45,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-19 07:55:45,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1779836429=10, srv213031447=16, srv1370108999=7, srv728512903=24, srv939392979=27, srv2062928799=15, srv1225432286=5, srv941264076=28, srv1220879565=4, srv827831503=26, srv623409017=23, srv98107085=29, srv157907148=8, srv1864592603=13, srv131989639=6, srv43285656=20, srv2533511=17, srv1168967785=2, srv1833968024=11, srv1174510867=3, srv1880297232=14, srv302754015=18, srv43335643=21, srv1076637379=1, srv1034407790=0, srv429942958=19, srv796544135=25, srv550313867=22, srv1739704627=9, srv1859297487=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-19 07:55:45,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-19 07:55:45,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-19 07:55:45,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-19 07:55:45,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-19 07:55:45,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-19 07:55:45,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-19 07:55:45,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-19 07:55:45,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-19 07:55:45,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-19 07:55:45,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-19 07:55:45,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-19 07:55:45,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-19 07:55:45,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-19 07:55:45,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-19 07:55:45,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-19 07:55:45,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-19 07:55:45,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-19 07:55:45,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-19 07:55:45,859 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.109016519104643, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-19 07:55:45,860 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912536550756261, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.109016519104643, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-04-19 07:57:45,849 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5719987 different iterations.  Found a solution that moves 1860 regions; Going from a computed imbalance of 0.9912536550756261 to a new imbalance of 4.341332789166274E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6888888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-19 07:57:45,854 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1034407790:90 , srv1076637379:90 , srv1168967785:90 , srv1174510867:90 , srv1220879565:90 , srv1225432286:90 , srv131989639:90 , srv1370108999:90 , srv157907148:90 , srv1739704627:90 , srv1779836429:90 , srv1833968024:90 , srv1859297487:90 , srv1864592603:90 , srv1880297232:90 , srv2062928799:90 , srv213031447:90 , srv2533511:90 , srv302754015:90 , srv429942958:90 , srv43285656:90 , srv43335643:90 , srv550313867:90 , srv623409017:90 , srv728512903:90 , srv796544135:90 , srv827831503:90 , srv939392979:90 , srv941264076:90 , srv98107085:90 }
2023-04-19 07:57:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-19 07:57:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1779836429=10, srv213031447=16, srv1370108999=7, srv728512903=24, srv939392979=27, srv2062928799=15, srv1225432286=5, srv941264076=28, srv1220879565=4, srv827831503=26, srv623409017=23, srv98107085=29, srv157907148=8, srv1864592603=13, srv131989639=6, srv43285656=20, srv2533511=17, srv1168967785=2, srv1833968024=11, srv1174510867=3, srv1880297232=14, srv302754015=18, srv43335643=21, srv1076637379=1, srv1034407790=0, srv429942958=19, srv796544135=25, srv550313867=22, srv1739704627=9, srv1859297487=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-19 07:57:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-19 07:57:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-19 07:57:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-19 07:57:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-19 07:57:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-19 07:57:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-19 07:57:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-19 07:57:45,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-19 07:57:45,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-19 07:57:45,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-19 07:57:45,877 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-19 07:57:45,878 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1034407790:90 , srv1076637379:90 , srv1168967785:90 , srv1174510867:90 , srv1220879565:90 , srv1225432286:90 , srv131989639:90 , srv1370108999:90 , srv157907148:90 , srv1739704627:90 , srv1779836429:90 , srv1833968024:90 , srv1859297487:90 , srv1864592603:90 , srv1880297232:90 , srv2062928799:90 , srv213031447:90 , srv2533511:90 , srv302754015:90 , srv429942958:90 , srv43285656:90 , srv43335643:90 , srv550313867:90 , srv623409017:90 , srv728512903:90 , srv796544135:90 , srv827831503:90 , srv939392979:90 , srv941264076:90 , srv98107085:90 }
2023-04-19 07:57:45,891 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=315 (was 493), ProcessCount=169 (was 169), AvailableMemoryMB=6846 (was 6598) - AvailableMemoryMB LEAK? -