2023-03-22 05:59:11,752 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-22 05:59:12,044 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-22 05:59:12,136 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-22 05:59:12,188 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-22 05:59:12,194 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-22 05:59:12,204 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=403, ProcessCount=167, AvailableMemoryMB=10016
2023-03-22 05:59:12,206 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-22 05:59:12,207 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-22 05:59:12,342 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1022756376:84 , srv1044766511:84 , srv1096919579:84 , srv1169305795:84 , srv1208064654:84 , srv122418914:84 , srv122854730:84 , srv1247907707:84 , srv1258787778:84 , srv1277022678:84 , srv1286848210:84 , srv1301673697:84 , srv1328751096:84 , srv1333247976:84 , srv1346274586:84 , srv1374162561:84 , srv143082455:84 , srv1463632293:84 , srv1488172339:84 , srv1492887753:84 , srv1501092102:84 , srv153166207:84 , srv1535863821:84 , srv1550875569:84 , srv1552358185:84 , srv159109524:84 , srv159639604:84 , srv1596600663:84 , srv1660015170:84 , srv1668550166:84 , srv1672596401:84 , srv1715179372:84 , srv1734983860:84 , srv1745449980:84 , srv1765944874:84 , srv1769094154:84 , srv1781042571:84 , srv1807433518:84 , srv1815610720:84 , srv1830565198:84 , srv1832553046:84 , srv1852845855:84 , srv1853428478:84 , srv1861841035:84 , srv1920406575:84 , srv1937154281:84 , srv1941479899:84 , srv1942745737:84 , srv1959045103:84 , srv1961979975:84 , srv200508400:84 , srv2029821037:84 , srv2041899969:84 , srv2085474430:84 , srv2095337098:84 , srv213336420:84 , srv2146539260:84 , srv215395085:84 , srv273906802:84 , srv281566561:84 , srv291208837:84 , srv297087262:84 , srv311099807:84 , srv319411385:84 , srv325290796:84 , srv330802636:84 , srv345602266:84 , srv355066946:84 , srv35855778:84 , srv360038049:84 , srv369143645:84 , srv375423374:84 , srv40411505:84 , srv444850088:84 , srv470123844:84 , srv498855385:84 , srv499928027:84 , srv511299883:84 , srv526969552:84 , srv568881357:84 , srv589908816:84 , srv59813157:84 , srv608307266:84 , srv638419604:84 , srv639066969:84 , srv67603957:84 , srv732376086:84 , srv757451465:84 , srv760938841:84 , srv786475324:84 , srv788760997:84 , srv818814686:84 , srv832765391:84 , srv844705011:84 , srv884144065:84 , srv887408197:84 , srv895734169:84 , srv962133720:84 , srv968364199:84 , srv1414174821:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-22 05:59:12,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 05:59:12,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1492887753=20, srv2029821037=52, srv1832553046=41, srv1258787778=8, srv369143645=71, srv638419604=84, srv962133720=98, srv968364199=99, srv297087262=62, srv325290796=65, srv470123844=75, srv1247907707=7, srv895734169=97, srv355066946=68, srv1830565198=40, srv122418914=5, srv1333247976=13, srv589908816=81, srv330802636=66, srv1463632293=18, srv526969552=79, srv1277022678=9, srv1852845855=42, srv1920406575=45, srv511299883=78, srv1301673697=11, srv1328751096=12, srv1672596401=31, srv1734983860=33, srv35855778=69, srv1286848210=10, srv1769094154=36, srv1961979975=50, srv159639604=27, srv1488172339=19, srv1815610720=39, srv311099807=63, srv1208064654=4, srv213336420=56, srv1535863821=23, srv200508400=51, srv1765944874=35, srv757451465=88, srv153166207=22, srv844705011=94, srv1552358185=25, srv159109524=26, srv444850088=74, srv1942745737=48, srv1937154281=46, srv1745449980=34, srv143082455=17, srv1550875569=24, srv1346274586=14, srv1668550166=30, srv608307266=83, srv1807433518=38, srv887408197=96, srv1596600663=28, srv281566561=60, srv215395085=58, srv1781042571=37, srv59813157=82, srv1853428478=43, srv1501092102=21, srv1660015170=29, srv1715179372=32, srv1374162561=15, srv2041899969=53, srv1096919579=2, srv2085474430=54, srv1022756376=0, srv1941479899=47, srv818814686=92, srv1959045103=49, srv360038049=70, srv273906802=59, srv2146539260=57, srv1414174821=16, srv639066969=85, srv788760997=91, srv786475324=90, srv2095337098=55, srv832765391=93, srv319411385=64, srv1169305795=3, srv568881357=80, srv1044766511=1, srv122854730=6, srv1861841035=44, srv498855385=76, srv67603957=86, srv40411505=73, srv884144065=95, srv375423374=72, srv732376086=87, srv291208837=61, srv499928027=77, srv760938841=89, srv345602266=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 05:59:12,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 05:59:12,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 05:59:12,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 05:59:12,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 05:59:12,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 05:59:12,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 05:59:12,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 05:59:12,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 05:59:12,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 05:59:12,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 05:59:12,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 05:59:12,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 05:59:12,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 05:59:12,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 05:59:12,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 05:59:12,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 05:59:12,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 05:59:12,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 05:59:12,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 05:59:12,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 05:59:12,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 05:59:12,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 05:59:12,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 05:59:12,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 05:59:12,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 05:59:12,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 05:59:12,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 05:59:12,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 05:59:12,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 05:59:12,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 05:59:12,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 05:59:12,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 05:59:12,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 05:59:12,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 05:59:12,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 05:59:12,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 05:59:12,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 05:59:12,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 05:59:12,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 05:59:12,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 05:59:12,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 05:59:12,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 05:59:12,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 05:59:12,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 05:59:12,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 05:59:12,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 05:59:12,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 05:59:12,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 05:59:12,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 05:59:12,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 05:59:12,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 05:59:12,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 05:59:12,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 05:59:12,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 05:59:12,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 05:59:12,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 05:59:12,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 05:59:12,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 05:59:12,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 05:59:12,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 05:59:12,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 05:59:12,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 05:59:12,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 05:59:12,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 05:59:12,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 05:59:12,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 05:59:12,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 05:59:12,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 05:59:12,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 05:59:12,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 05:59:12,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 05:59:12,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 05:59:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 05:59:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 05:59:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 05:59:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 05:59:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 05:59:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 05:59:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 05:59:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 05:59:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 05:59:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 05:59:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 05:59:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 05:59:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 05:59:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 05:59:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 05:59:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 05:59:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 05:59:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 05:59:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 05:59:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 05:59:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 05:59:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 05:59:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 05:59:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 05:59:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 05:59:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 05:59:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 05:59:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 05:59:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 05:59:12,653 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 05:59:12,654 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-22 05:59:12,654 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-22 05:59:12,941 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 450 ms to try 5000 different iterations.  Found a solution that moves 4658 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18527149784671013. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03996693982328094, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05712726258268285, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5175555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03996693982328094, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18033333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24925, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 05:59:12,957 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1715179372:81 , srv67603957:81 , srv215395085:82 , srv444850088:82 , srv470123844:82 , srv498855385:82 , srv832765391:82 , srv1333247976:83 , srv143082455:83 , srv1501092102:83 , srv1535863821:83 , srv1832553046:83 , srv1942745737:83 , srv638419604:83 , srv760938841:83 , srv122418914:84 , srv1247907707:84 , srv1374162561:84 , srv1660015170:84 , srv1672596401:84 , srv1745449980:84 , srv1959045103:84 , srv200508400:84 , srv2029821037:84 , srv2041899969:84 , srv40411505:84 , srv786475324:84 , srv844705011:84 , srv1208064654:85 , srv1258787778:85 , srv1769094154:85 , srv1815610720:85 , srv273906802:85 , srv291208837:85 , srv732376086:85 , srv1044766511:86 , srv1463632293:86 , srv1550875569:86 , srv1596600663:86 , srv1668550166:86 , srv1807433518:86 , srv1852845855:86 , srv1853428478:86 , srv1920406575:86 , srv1937154281:86 , srv2085474430:86 , srv281566561:86 , srv297087262:86 , srv325290796:86 , srv35855778:86 , srv369143645:86 , srv59813157:86 , srv757451465:86 , srv818814686:86 , srv1022756376:87 , srv1096919579:87 , srv1169305795:87 , srv1346274586:87 , srv1552358185:87 , srv1830565198:87 , srv1861841035:87 , srv2095337098:87 , srv213336420:87 , srv311099807:87 , srv499928027:87 , srv511299883:87 , srv589908816:87 , srv639066969:87 , srv788760997:87 , srv884144065:87 , srv1328751096:88 , srv153166207:88 , srv159109524:88 , srv1765944874:88 , srv319411385:88 , srv330802636:88 , srv526969552:88 , srv887408197:88 , srv895734169:88 , srv962133720:88 , srv968364199:88 , srv1277022678:89 , srv1781042571:89 , srv1941479899:89 , srv1734983860:90 , srv1961979975:90 , srv345602266:90 , srv355066946:90 , srv608307266:90 , srv1286848210:91 , srv1301673697:91 , srv1488172339:91 , srv568881357:91 , srv159639604:92 , srv360038049:92 , srv122854730:93 , srv1492887753:93 , srv375423374:93 , srv2146539260:94 , srv1414174821:445 }
2023-03-22 05:59:12,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 05:59:12,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1492887753=20, srv2029821037=52, srv1832553046=41, srv1258787778=8, srv369143645=71, srv638419604=84, srv962133720=98, srv968364199=99, srv297087262=62, srv325290796=65, srv470123844=75, srv1247907707=7, srv895734169=97, srv355066946=68, srv1830565198=40, srv122418914=5, srv1333247976=13, srv589908816=81, srv330802636=66, srv1463632293=18, srv526969552=79, srv1277022678=9, srv1852845855=42, srv1920406575=45, srv511299883=78, srv1301673697=11, srv1328751096=12, srv1672596401=31, srv1734983860=33, srv35855778=69, srv1286848210=10, srv1769094154=36, srv1961979975=50, srv159639604=27, srv1488172339=19, srv1815610720=39, srv311099807=63, srv1208064654=4, srv213336420=56, srv1535863821=23, srv200508400=51, srv1765944874=35, srv757451465=88, srv153166207=22, srv844705011=94, srv1552358185=25, srv159109524=26, srv444850088=74, srv1942745737=48, srv1937154281=46, srv1745449980=34, srv143082455=17, srv1550875569=24, srv1346274586=14, srv1668550166=30, srv608307266=83, srv1807433518=38, srv887408197=96, srv1596600663=28, srv281566561=60, srv215395085=58, srv1781042571=37, srv59813157=82, srv1853428478=43, srv1501092102=21, srv1660015170=29, srv1715179372=32, srv1374162561=15, srv2041899969=53, srv1096919579=2, srv2085474430=54, srv1022756376=0, srv1941479899=47, srv818814686=92, srv1959045103=49, srv360038049=70, srv273906802=59, srv2146539260=57, srv1414174821=16, srv639066969=85, srv788760997=91, srv786475324=90, srv2095337098=55, srv832765391=93, srv319411385=64, srv1169305795=3, srv568881357=80, srv1044766511=1, srv122854730=6, srv1861841035=44, srv498855385=76, srv67603957=86, srv40411505=73, srv884144065=95, srv375423374=72, srv732376086=87, srv291208837=61, srv499928027=77, srv760938841=89, srv345602266=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 05:59:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 05:59:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 05:59:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 05:59:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 05:59:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 05:59:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 05:59:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 05:59:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 05:59:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 05:59:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 05:59:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 05:59:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 05:59:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 05:59:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 05:59:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 05:59:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 05:59:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 05:59:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 05:59:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 05:59:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 05:59:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 05:59:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 05:59:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 05:59:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 05:59:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 05:59:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 05:59:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 05:59:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 05:59:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 05:59:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 05:59:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 05:59:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 05:59:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 05:59:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 05:59:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 05:59:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 05:59:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 05:59:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 05:59:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 05:59:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 05:59:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 05:59:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 05:59:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 05:59:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 05:59:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 05:59:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 05:59:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 05:59:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 05:59:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 05:59:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 05:59:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 05:59:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 05:59:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 05:59:12,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 05:59:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 05:59:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 05:59:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 05:59:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 05:59:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 05:59:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 05:59:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 05:59:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 05:59:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 05:59:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 05:59:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 05:59:13,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 05:59:13,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 05:59:13,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 05:59:13,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 05:59:13,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 05:59:13,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 05:59:13,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 05:59:13,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 05:59:13,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 05:59:13,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 05:59:13,110 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03996693982328094, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05712726258268285, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03996693982328094, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18033333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24925, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 05:59:13,110 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-22 05:59:13,111 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18523888183359413, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03996693982328094, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05712726258268285, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03996693982328094, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18033333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24925, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-22 05:59:13,291 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 264 ms to try 5000 different iterations.  Found a solution that moves 2670 regions; Going from a computed imbalance of 0.18523888183359413 to a new imbalance of 0.016500804388282518. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0207106563920738); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036072629068328727, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2966666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0207106563920738); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04933333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 05:59:13,307 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2146539260:77 , srv1492887753:78 , srv1941479899:79 , srv375423374:80 , srv1286848210:81 , srv1488172339:81 , srv159639604:81 , srv1301673697:82 , srv1734983860:82 , srv355066946:82 , srv360038049:82 , srv608307266:82 , srv1328751096:83 , srv153166207:83 , srv1765944874:83 , srv2085474430:83 , srv213336420:83 , srv319411385:83 , srv526969552:83 , srv568881357:83 , srv757451465:83 , srv1096919579:84 , srv122854730:84 , srv1277022678:84 , srv1781042571:84 , srv1961979975:84 , srv2095337098:84 , srv330802636:84 , srv345602266:84 , srv884144065:84 , srv887408197:84 , srv962133720:84 , srv968364199:84 , srv1169305795:85 , srv1346274586:85 , srv1463632293:85 , srv1745449980:85 , srv1830565198:85 , srv273906802:85 , srv311099807:85 , srv639066969:85 , srv895734169:85 , srv1044766511:86 , srv1258787778:86 , srv143082455:86 , srv1550875569:86 , srv1552358185:86 , srv159109524:86 , srv1596600663:86 , srv1668550166:86 , srv1769094154:86 , srv1920406575:86 , srv1937154281:86 , srv297087262:86 , srv511299883:86 , srv589908816:86 , srv59813157:86 , srv732376086:86 , srv788760997:86 , srv818814686:86 , srv1022756376:87 , srv122418914:87 , srv1660015170:87 , srv1672596401:87 , srv1807433518:87 , srv1852845855:87 , srv1861841035:87 , srv1959045103:87 , srv281566561:87 , srv325290796:87 , srv35855778:87 , srv369143645:87 , srv40411505:87 , srv499928027:87 , srv1208064654:88 , srv1815610720:88 , srv1853428478:88 , srv2041899969:88 , srv638419604:88 , srv786475324:88 , srv1247907707:89 , srv1374162561:89 , srv1535863821:89 , srv1832553046:89 , srv1942745737:89 , srv200508400:89 , srv215395085:89 , srv291208837:89 , srv498855385:89 , srv760938841:89 , srv844705011:89 , srv1333247976:90 , srv1715179372:90 , srv470123844:90 , srv832765391:90 , srv1501092102:91 , srv67603957:91 , srv2029821037:92 , srv444850088:93 , srv1414174821:513 }
2023-03-22 05:59:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 05:59:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1492887753=20, srv2029821037=52, srv1832553046=41, srv1258787778=8, srv369143645=71, srv638419604=84, srv962133720=98, srv968364199=99, srv297087262=62, srv325290796=65, srv470123844=75, srv1247907707=7, srv895734169=97, srv355066946=68, srv1830565198=40, srv122418914=5, srv1333247976=13, srv589908816=81, srv330802636=66, srv1463632293=18, srv526969552=79, srv1277022678=9, srv1852845855=42, srv1920406575=45, srv511299883=78, srv1301673697=11, srv1328751096=12, srv1672596401=31, srv1734983860=33, srv35855778=69, srv1286848210=10, srv1769094154=36, srv1961979975=50, srv159639604=27, srv1488172339=19, srv1815610720=39, srv311099807=63, srv1208064654=4, srv213336420=56, srv1535863821=23, srv200508400=51, srv1765944874=35, srv757451465=88, srv153166207=22, srv844705011=94, srv1552358185=25, srv159109524=26, srv444850088=74, srv1942745737=48, srv1937154281=46, srv1745449980=34, srv143082455=17, srv1550875569=24, srv1346274586=14, srv1668550166=30, srv608307266=83, srv1807433518=38, srv887408197=96, srv1596600663=28, srv281566561=60, srv215395085=58, srv1781042571=37, srv59813157=82, srv1853428478=43, srv1501092102=21, srv1660015170=29, srv1715179372=32, srv1374162561=15, srv2041899969=53, srv1096919579=2, srv2085474430=54, srv1022756376=0, srv1941479899=47, srv818814686=92, srv1959045103=49, srv360038049=70, srv273906802=59, srv2146539260=57, srv1414174821=16, srv639066969=85, srv788760997=91, srv786475324=90, srv2095337098=55, srv832765391=93, srv319411385=64, srv1169305795=3, srv568881357=80, srv1044766511=1, srv122854730=6, srv1861841035=44, srv498855385=76, srv67603957=86, srv40411505=73, srv884144065=95, srv375423374=72, srv732376086=87, srv291208837=61, srv499928027=77, srv760938841=89, srv345602266=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 05:59:13,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 05:59:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 05:59:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 05:59:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 05:59:13,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 05:59:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 05:59:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 05:59:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 05:59:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 05:59:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 05:59:13,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 05:59:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 05:59:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 05:59:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 05:59:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 05:59:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 05:59:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 05:59:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 05:59:13,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 05:59:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 05:59:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 05:59:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 05:59:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 05:59:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 05:59:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 05:59:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 05:59:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 05:59:13,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 05:59:13,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 05:59:13,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 05:59:13,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 05:59:13,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 05:59:13,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 05:59:13,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 05:59:13,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 05:59:13,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 05:59:13,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 05:59:13,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 05:59:13,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 05:59:13,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 05:59:13,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 05:59:13,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 05:59:13,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 05:59:13,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 05:59:13,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 05:59:13,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 05:59:13,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 05:59:13,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 05:59:13,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 05:59:13,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 05:59:13,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 05:59:13,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 05:59:13,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 05:59:13,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 05:59:13,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 05:59:13,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 05:59:13,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 05:59:13,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 05:59:13,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 05:59:13,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 05:59:13,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 05:59:13,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 05:59:13,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 05:59:13,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 05:59:13,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 05:59:13,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 05:59:13,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 05:59:13,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 05:59:13,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 05:59:13,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 05:59:13,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 05:59:13,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 05:59:13,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 05:59:13,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 05:59:13,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 05:59:13,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 05:59:13,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 05:59:13,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 05:59:13,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 05:59:13,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 05:59:13,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 05:59:13,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 05:59:13,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 05:59:13,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 05:59:13,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 05:59:13,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 05:59:13,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 05:59:13,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 05:59:13,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 05:59:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 05:59:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 05:59:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 05:59:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 05:59:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 05:59:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 05:59:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 05:59:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 05:59:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 05:59:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 05:59:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 05:59:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 05:59:13,480 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0207106563920738); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036072629068328727, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0207106563920738); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04933333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 05:59:13,480 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-22 05:59:13,480 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016482108648690465, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0207106563920738); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036072629068328727, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0207106563920738); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04933333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-22 05:59:13,681 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 310 ms to try 5000 different iterations.  Found a solution that moves 1074 regions; Going from a computed imbalance of 0.016482108648690465 to a new imbalance of 8.144486488684473E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004240553272435818); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006391633139214696); MoveCostFunction : (multiplier=7.0, imbalance=0.11933333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004240553272435818); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 05:59:13,682 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv122854730:81 , srv2029821037:82 , srv1374162561:83 , srv1861841035:83 , srv1961979975:83 , srv345602266:83 , srv444850088:83 , srv568881357:83 , srv1022756376:84 , srv1208064654:84 , srv1501092102:84 , srv159109524:84 , srv159639604:84 , srv1781042571:84 , srv1807433518:84 , srv1815610720:84 , srv1852845855:84 , srv281566561:84 , srv35855778:84 , srv360038049:84 , srv369143645:84 , srv375423374:84 , srv499928027:84 , srv511299883:84 , srv589908816:84 , srv67603957:84 , srv844705011:84 , srv1044766511:85 , srv1247907707:85 , srv1301673697:85 , srv1333247976:85 , srv1346274586:85 , srv1488172339:85 , srv1535863821:85 , srv1552358185:85 , srv1596600663:85 , srv1830565198:85 , srv1853428478:85 , srv1937154281:85 , srv1942745737:85 , srv200508400:85 , srv291208837:85 , srv297087262:85 , srv311099807:85 , srv355066946:85 , srv608307266:85 , srv786475324:85 , srv788760997:85 , srv832765391:85 , srv887408197:85 , srv968364199:85 , srv1096919579:86 , srv1169305795:86 , srv122418914:86 , srv1277022678:86 , srv1286848210:86 , srv1463632293:86 , srv153166207:86 , srv1550875569:86 , srv1660015170:86 , srv1668550166:86 , srv1672596401:86 , srv1715179372:86 , srv1734983860:86 , srv1832553046:86 , srv1920406575:86 , srv319411385:86 , srv330802636:86 , srv470123844:86 , srv526969552:86 , srv59813157:86 , srv638419604:86 , srv760938841:86 , srv818814686:86 , srv1258787778:87 , srv1328751096:87 , srv1765944874:87 , srv1769094154:87 , srv1959045103:87 , srv2041899969:87 , srv2095337098:87 , srv213336420:87 , srv2146539260:87 , srv325290796:87 , srv40411505:87 , srv639066969:87 , srv732376086:87 , srv884144065:87 , srv895734169:87 , srv962133720:87 , srv143082455:88 , srv1492887753:88 , srv215395085:88 , srv273906802:88 , srv498855385:88 , srv757451465:88 , srv1745449980:89 , srv1941479899:89 , srv2085474430:90 , srv1414174821:537 }
2023-03-22 05:59:13,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 05:59:13,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1492887753=20, srv2029821037=52, srv1832553046=41, srv1258787778=8, srv369143645=71, srv638419604=84, srv962133720=98, srv968364199=99, srv297087262=62, srv325290796=65, srv470123844=75, srv1247907707=7, srv895734169=97, srv355066946=68, srv1830565198=40, srv122418914=5, srv1333247976=13, srv589908816=81, srv330802636=66, srv1463632293=18, srv526969552=79, srv1277022678=9, srv1852845855=42, srv1920406575=45, srv511299883=78, srv1301673697=11, srv1328751096=12, srv1672596401=31, srv1734983860=33, srv35855778=69, srv1286848210=10, srv1769094154=36, srv1961979975=50, srv159639604=27, srv1488172339=19, srv1815610720=39, srv311099807=63, srv1208064654=4, srv213336420=56, srv1535863821=23, srv200508400=51, srv1765944874=35, srv757451465=88, srv153166207=22, srv844705011=94, srv1552358185=25, srv159109524=26, srv444850088=74, srv1942745737=48, srv1937154281=46, srv1745449980=34, srv143082455=17, srv1550875569=24, srv1346274586=14, srv1668550166=30, srv608307266=83, srv1807433518=38, srv887408197=96, srv1596600663=28, srv281566561=60, srv215395085=58, srv1781042571=37, srv59813157=82, srv1853428478=43, srv1501092102=21, srv1660015170=29, srv1715179372=32, srv1374162561=15, srv2041899969=53, srv1096919579=2, srv2085474430=54, srv1022756376=0, srv1941479899=47, srv818814686=92, srv1959045103=49, srv360038049=70, srv273906802=59, srv2146539260=57, srv1414174821=16, srv639066969=85, srv788760997=91, srv786475324=90, srv2095337098=55, srv832765391=93, srv319411385=64, srv1169305795=3, srv568881357=80, srv1044766511=1, srv122854730=6, srv1861841035=44, srv498855385=76, srv67603957=86, srv40411505=73, srv884144065=95, srv375423374=72, srv732376086=87, srv291208837=61, srv499928027=77, srv760938841=89, srv345602266=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 05:59:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 05:59:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 05:59:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 05:59:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 05:59:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 05:59:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 05:59:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 05:59:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 05:59:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 05:59:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 05:59:13,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 05:59:13,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 05:59:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 05:59:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 05:59:13,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 05:59:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 05:59:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 05:59:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 05:59:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 05:59:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 05:59:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 05:59:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 05:59:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 05:59:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 05:59:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 05:59:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 05:59:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 05:59:13,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 05:59:13,793 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004240553272435818); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006391633139214696); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004240553272435818); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 05:59:13,793 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-22 05:59:13,793 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.069283401336656E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004240553272435818); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006391633139214696); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004240553272435818); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-22 05:59:14,096 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 388 ms to try 5000 different iterations.  Found a solution that moves 243 regions; Going from a computed imbalance of 8.069283401336656E-4 to a new imbalance of 2.090526786619967E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.027, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 05:59:14,098 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv325290796:82 , srv895734169:82 , srv1277022678:83 , srv1328751096:83 , srv1492887753:83 , srv2085474430:83 , srv215395085:83 , srv291208837:83 , srv498855385:83 , srv639066969:83 , srv962133720:83 , srv1022756376:84 , srv1169305795:84 , srv1208064654:84 , srv122854730:84 , srv1247907707:84 , srv1258787778:84 , srv1286848210:84 , srv1301673697:84 , srv1501092102:84 , srv1550875569:84 , srv1552358185:84 , srv159109524:84 , srv1668550166:84 , srv1715179372:84 , srv1734983860:84 , srv1745449980:84 , srv1769094154:84 , srv1832553046:84 , srv1853428478:84 , srv1942745737:84 , srv1959045103:84 , srv200508400:84 , srv2029821037:84 , srv2041899969:84 , srv2095337098:84 , srv213336420:84 , srv2146539260:84 , srv273906802:84 , srv281566561:84 , srv319411385:84 , srv330802636:84 , srv40411505:84 , srv444850088:84 , srv470123844:84 , srv499928027:84 , srv568881357:84 , srv59813157:84 , srv638419604:84 , srv732376086:84 , srv760938841:84 , srv788760997:84 , srv818814686:84 , srv884144065:84 , srv887408197:84 , srv1044766511:85 , srv1096919579:85 , srv122418914:85 , srv1333247976:85 , srv1346274586:85 , srv143082455:85 , srv1463632293:85 , srv153166207:85 , srv1535863821:85 , srv159639604:85 , srv1596600663:85 , srv1660015170:85 , srv1672596401:85 , srv1765944874:85 , srv1781042571:85 , srv1807433518:85 , srv1815610720:85 , srv1830565198:85 , srv1852845855:85 , srv1861841035:85 , srv1920406575:85 , srv1937154281:85 , srv1961979975:85 , srv311099807:85 , srv345602266:85 , srv355066946:85 , srv35855778:85 , srv360038049:85 , srv375423374:85 , srv511299883:85 , srv526969552:85 , srv589908816:85 , srv608307266:85 , srv757451465:85 , srv786475324:85 , srv832765391:85 , srv844705011:85 , srv968364199:85 , srv1374162561:86 , srv1488172339:86 , srv1941479899:86 , srv297087262:86 , srv369143645:86 , srv67603957:86 , srv1414174821:647 }
2023-03-22 05:59:14,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 05:59:14,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1492887753=20, srv2029821037=52, srv1832553046=41, srv1258787778=8, srv369143645=71, srv638419604=84, srv962133720=98, srv968364199=99, srv297087262=62, srv325290796=65, srv470123844=75, srv1247907707=7, srv895734169=97, srv355066946=68, srv1830565198=40, srv122418914=5, srv1333247976=13, srv589908816=81, srv330802636=66, srv1463632293=18, srv526969552=79, srv1277022678=9, srv1852845855=42, srv1920406575=45, srv511299883=78, srv1301673697=11, srv1328751096=12, srv1672596401=31, srv1734983860=33, srv35855778=69, srv1286848210=10, srv1769094154=36, srv1961979975=50, srv159639604=27, srv1488172339=19, srv1815610720=39, srv311099807=63, srv1208064654=4, srv213336420=56, srv1535863821=23, srv200508400=51, srv1765944874=35, srv757451465=88, srv153166207=22, srv844705011=94, srv1552358185=25, srv159109524=26, srv444850088=74, srv1942745737=48, srv1937154281=46, srv1745449980=34, srv143082455=17, srv1550875569=24, srv1346274586=14, srv1668550166=30, srv608307266=83, srv1807433518=38, srv887408197=96, srv1596600663=28, srv281566561=60, srv215395085=58, srv1781042571=37, srv59813157=82, srv1853428478=43, srv1501092102=21, srv1660015170=29, srv1715179372=32, srv1374162561=15, srv2041899969=53, srv1096919579=2, srv2085474430=54, srv1022756376=0, srv1941479899=47, srv818814686=92, srv1959045103=49, srv360038049=70, srv273906802=59, srv2146539260=57, srv1414174821=16, srv639066969=85, srv788760997=91, srv786475324=90, srv2095337098=55, srv832765391=93, srv319411385=64, srv1169305795=3, srv568881357=80, srv1044766511=1, srv122854730=6, srv1861841035=44, srv498855385=76, srv67603957=86, srv40411505=73, srv884144065=95, srv375423374=72, srv732376086=87, srv291208837=61, srv499928027=77, srv760938841=89, srv345602266=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 05:59:14,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 05:59:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 05:59:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 05:59:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 05:59:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 05:59:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 05:59:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 05:59:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 05:59:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 05:59:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 05:59:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 05:59:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 05:59:14,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 05:59:14,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 05:59:14,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 05:59:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 05:59:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 05:59:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 05:59:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 05:59:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 05:59:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 05:59:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 05:59:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 05:59:14,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 05:59:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 05:59:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 05:59:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 05:59:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 05:59:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 05:59:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 05:59:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 05:59:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 05:59:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 05:59:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 05:59:14,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 05:59:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 05:59:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 05:59:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 05:59:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 05:59:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 05:59:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 05:59:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 05:59:14,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 05:59:14,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 05:59:14,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 05:59:14,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 05:59:14,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 05:59:14,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 05:59:14,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 05:59:14,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 05:59:14,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 05:59:14,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 05:59:14,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 05:59:14,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 05:59:14,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 05:59:14,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 05:59:14,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 05:59:14,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 05:59:14,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 05:59:14,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 05:59:14,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 05:59:14,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 05:59:14,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 05:59:14,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 05:59:14,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 05:59:14,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 05:59:14,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 05:59:14,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 05:59:14,204 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.920374549883289E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 05:59:14,205 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv325290796:82 , srv895734169:82 , srv1277022678:83 , srv1328751096:83 , srv1492887753:83 , srv2085474430:83 , srv215395085:83 , srv291208837:83 , srv498855385:83 , srv639066969:83 , srv962133720:83 , srv1022756376:84 , srv1169305795:84 , srv1208064654:84 , srv122854730:84 , srv1247907707:84 , srv1258787778:84 , srv1286848210:84 , srv1301673697:84 , srv1501092102:84 , srv1550875569:84 , srv1552358185:84 , srv159109524:84 , srv1668550166:84 , srv1715179372:84 , srv1734983860:84 , srv1745449980:84 , srv1769094154:84 , srv1832553046:84 , srv1853428478:84 , srv1942745737:84 , srv1959045103:84 , srv200508400:84 , srv2029821037:84 , srv2041899969:84 , srv2095337098:84 , srv213336420:84 , srv2146539260:84 , srv273906802:84 , srv281566561:84 , srv319411385:84 , srv330802636:84 , srv40411505:84 , srv444850088:84 , srv470123844:84 , srv499928027:84 , srv568881357:84 , srv59813157:84 , srv638419604:84 , srv732376086:84 , srv760938841:84 , srv788760997:84 , srv818814686:84 , srv884144065:84 , srv887408197:84 , srv1044766511:85 , srv1096919579:85 , srv122418914:85 , srv1333247976:85 , srv1346274586:85 , srv143082455:85 , srv1463632293:85 , srv153166207:85 , srv1535863821:85 , srv159639604:85 , srv1596600663:85 , srv1660015170:85 , srv1672596401:85 , srv1765944874:85 , srv1781042571:85 , srv1807433518:85 , srv1815610720:85 , srv1830565198:85 , srv1852845855:85 , srv1861841035:85 , srv1920406575:85 , srv1937154281:85 , srv1961979975:85 , srv311099807:85 , srv345602266:85 , srv355066946:85 , srv35855778:85 , srv360038049:85 , srv375423374:85 , srv511299883:85 , srv526969552:85 , srv589908816:85 , srv608307266:85 , srv757451465:85 , srv786475324:85 , srv832765391:85 , srv844705011:85 , srv968364199:85 , srv1374162561:86 , srv1488172339:86 , srv1941479899:86 , srv297087262:86 , srv369143645:86 , srv67603957:86 , srv1414174821:647 }
2023-03-22 05:59:14,227 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=403 (was 403), ProcessCount=167 (was 167), AvailableMemoryMB=9508 (was 10016)
2023-03-22 05:59:14,233 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=403, ProcessCount=167, AvailableMemoryMB=9508
2023-03-22 05:59:14,234 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-22 05:59:14,234 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-22 05:59:14,241 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1090705130:84 , srv1100364431:84 , srv113593577:84 , srv1337737538:84 , srv1387769810:84 , srv1395259022:84 , srv1465648477:84 , srv1681577366:84 , srv1746561190:84 , srv1824104772:84 , srv1879534902:84 , srv1914406983:84 , srv1921165877:84 , srv1950811401:84 , srv2075576679:84 , srv2109478810:84 , srv212400472:84 , srv2138676947:84 , srv319554778:84 , srv338552471:84 , srv388009904:84 , srv393487680:84 , srv45615755:84 , srv530907842:84 , srv614194647:84 , srv692572540:84 , srv865481811:84 , srv901079450:84 , srv970448978:84 , srv917664400:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-22 05:59:14,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 05:59:14,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1879534902=10, srv692572540=25, srv530907842=23, srv1746561190=8, srv614194647=24, srv917664400=28, srv1387769810=4, srv2109478810=15, srv1090705130=0, srv1681577366=7, srv1950811401=13, srv113593577=2, srv1100364431=1, srv393487680=21, srv1824104772=9, srv45615755=22, srv865481811=26, srv388009904=20, srv970448978=29, srv1465648477=6, srv1921165877=12, srv901079450=27, srv2138676947=17, srv338552471=19, srv212400472=16, srv1337737538=3, srv1395259022=5, srv1914406983=11, srv319554778=18, srv2075576679=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 05:59:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 05:59:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 05:59:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 05:59:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 05:59:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 05:59:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 05:59:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 05:59:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 05:59:14,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 05:59:14,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 05:59:14,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 05:59:14,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 05:59:14,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 05:59:14,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 05:59:14,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 05:59:14,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 05:59:14,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 05:59:14,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-22 05:59:14,266 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0955254855597214, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 05:59:14,266 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912494040950085, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0955254855597214, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-22 06:01:14,255 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9743268 different iterations.  Found a solution that moves 1853 regions; Going from a computed imbalance of 0.9912494040950085 to a new imbalance of 4.324994439959734E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6862962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:01:14,260 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1090705130:90 , srv1100364431:90 , srv113593577:90 , srv1337737538:90 , srv1387769810:90 , srv1395259022:90 , srv1465648477:90 , srv1681577366:90 , srv1746561190:90 , srv1824104772:90 , srv1879534902:90 , srv1914406983:90 , srv1921165877:90 , srv1950811401:90 , srv2075576679:90 , srv2109478810:90 , srv212400472:90 , srv2138676947:90 , srv319554778:90 , srv338552471:90 , srv388009904:90 , srv393487680:90 , srv45615755:90 , srv530907842:90 , srv614194647:90 , srv692572540:90 , srv865481811:90 , srv901079450:90 , srv917664400:90 , srv970448978:90 }
2023-03-22 06:01:14,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 06:01:14,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1879534902=10, srv692572540=25, srv530907842=23, srv1746561190=8, srv614194647=24, srv917664400=28, srv1387769810=4, srv2109478810=15, srv1090705130=0, srv1681577366=7, srv1950811401=13, srv113593577=2, srv1100364431=1, srv393487680=21, srv1824104772=9, srv45615755=22, srv865481811=26, srv388009904=20, srv970448978=29, srv1465648477=6, srv1921165877=12, srv901079450=27, srv2138676947=17, srv338552471=19, srv212400472=16, srv1337737538=3, srv1395259022=5, srv1914406983=11, srv319554778=18, srv2075576679=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 06:01:14,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 06:01:14,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 06:01:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 06:01:14,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 06:01:14,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 06:01:14,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 06:01:14,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 06:01:14,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 06:01:14,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 06:01:14,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 06:01:14,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-22 06:01:14,274 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 06:01:14,274 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1090705130:90 , srv1100364431:90 , srv113593577:90 , srv1337737538:90 , srv1387769810:90 , srv1395259022:90 , srv1465648477:90 , srv1681577366:90 , srv1746561190:90 , srv1824104772:90 , srv1879534902:90 , srv1914406983:90 , srv1921165877:90 , srv1950811401:90 , srv2075576679:90 , srv2109478810:90 , srv212400472:90 , srv2138676947:90 , srv319554778:90 , srv338552471:90 , srv388009904:90 , srv393487680:90 , srv45615755:90 , srv530907842:90 , srv614194647:90 , srv692572540:90 , srv865481811:90 , srv901079450:90 , srv917664400:90 , srv970448978:90 }
2023-03-22 06:01:14,285 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=200 (was 403), ProcessCount=163 (was 167), AvailableMemoryMB=10159 (was 9508) - AvailableMemoryMB LEAK? -