2023-04-27 03:55:39,382 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-04-27 03:55:39,776 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-04-27 03:55:39,903 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-04-27 03:55:39,971 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-27 03:55:39,978 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-27 03:55:39,993 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=329, ProcessCount=170, AvailableMemoryMB=12771
2023-04-27 03:55:39,996 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-27 03:55:39,996 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-27 03:55:40,160 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004668178:84 , srv1010882045:84 , srv1089809605:84 , srv1094325529:84 , srv1100414175:84 , srv1142559099:84 , srv115458547:84 , srv1216645669:84 , srv1221873958:84 , srv1227722619:84 , srv1247106695:84 , srv1251297955:84 , srv125527346:84 , srv125591564:84 , srv1300332418:84 , srv1417857541:84 , srv1443009817:84 , srv1444218298:84 , srv1462411611:84 , srv1463294836:84 , srv1466536474:84 , srv1467188500:84 , srv153562850:84 , srv1538739766:84 , srv1539312760:84 , srv1541552339:84 , srv1543442543:84 , srv1545321396:84 , srv1581140820:84 , srv1640616532:84 , srv1652022331:84 , srv1677269974:84 , srv1677469211:84 , srv1704132265:84 , srv1707891155:84 , srv1737479722:84 , srv1754191090:84 , srv1755336894:84 , srv1799369018:84 , srv181611347:84 , srv1909649209:84 , srv1927983897:84 , srv1941893105:84 , srv1962043673:84 , srv1971714216:84 , srv1986981316:84 , srv1991713604:84 , srv1996656467:84 , srv200158938:84 , srv2028969550:84 , srv203808242:84 , srv2040347569:84 , srv206989088:84 , srv2089377918:84 , srv2098264623:84 , srv2121842066:84 , srv218430266:84 , srv223116909:84 , srv237300623:84 , srv29504339:84 , srv310511489:84 , srv324488649:84 , srv328238410:84 , srv343084686:84 , srv368477409:84 , srv37362796:84 , srv378045804:84 , srv38720932:84 , srv392718648:84 , srv398707149:84 , srv406524753:84 , srv426148073:84 , srv437286198:84 , srv496108236:84 , srv503762364:84 , srv506707850:84 , srv548488761:84 , srv55068605:84 , srv579509762:84 , srv595014969:84 , srv616022491:84 , srv616172402:84 , srv617399575:84 , srv630915047:84 , srv635326073:84 , srv667924310:84 , srv680500438:84 , srv681143533:84 , srv73556177:84 , srv776493440:84 , srv810111236:84 , srv833388346:84 , srv923290372:84 , srv933450361:84 , srv937241083:84 , srv941585437:84 , srv955226935:84 , srv97231620:84 , srv991491216:84 , srv2034671227:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-04-27 03:55:40,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-27 03:55:40,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1581140820=28, srv1754191090=36, srv37362796=66, srv328238410=63, srv398707149=70, srv1216645669=7, srv1909649209=40, srv2034671227=50, srv125527346=12, srv1100414175=4, srv680500438=87, srv1417857541=15, srv1704132265=33, srv153562850=22, srv1677469211=32, srv1010882045=1, srv506707850=76, srv630915047=84, srv616022491=81, srv616172402=82, srv833388346=92, srv1463294836=19, srv810111236=91, srv1247106695=10, srv125591564=13, srv1462411611=18, srv503762364=75, srv1227722619=9, srv1927983897=41, srv955226935=97, srv29504339=60, srv406524753=71, srv1545321396=27, srv310511489=61, srv426148073=72, srv1677269974=31, srv55068605=78, srv115458547=6, srv1089809605=2, srv181611347=39, srv1466536474=20, srv941585437=96, srv1996656467=47, srv1971714216=44, srv937241083=95, srv776493440=90, srv1799369018=38, srv1991713604=46, srv496108236=74, srv1443009817=16, srv200158938=48, srv2040347569=52, srv579509762=79, srv2121842066=56, srv203808242=51, srv206989088=53, srv548488761=77, srv378045804=67, srv667924310=86, srv2089377918=54, srv1962043673=43, srv1538739766=23, srv635326073=85, srv991491216=99, srv1467188500=21, srv923290372=93, srv2098264623=55, srv1652022331=30, srv324488649=62, srv1251297955=11, srv2028969550=49, srv343084686=64, srv392718648=69, srv1986981316=45, srv933450361=94, srv97231620=98, srv218430266=57, srv1300332418=14, srv1541552339=25, srv1444218298=17, srv1941893105=42, srv1640616532=29, srv1004668178=0, srv237300623=59, srv1539312760=24, srv223116909=58, srv1221873958=8, srv1142559099=5, srv437286198=73, srv368477409=65, srv73556177=89, srv1755336894=37, srv617399575=83, srv595014969=80, srv681143533=88, srv1737479722=35, srv1707891155=34, srv38720932=68, srv1094325529=3, srv1543442543=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-27 03:55:40,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-27 03:55:40,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-27 03:55:40,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-27 03:55:40,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-27 03:55:40,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-27 03:55:40,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-27 03:55:40,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-27 03:55:40,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-27 03:55:40,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-27 03:55:40,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-27 03:55:40,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-27 03:55:40,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-27 03:55:40,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-27 03:55:40,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-27 03:55:40,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-27 03:55:40,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-27 03:55:40,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-27 03:55:40,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-27 03:55:40,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-27 03:55:40,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-27 03:55:40,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-27 03:55:40,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-27 03:55:40,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-27 03:55:40,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-27 03:55:40,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-27 03:55:40,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-27 03:55:40,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-27 03:55:40,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-27 03:55:40,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-27 03:55:40,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-27 03:55:40,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-27 03:55:40,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-27 03:55:40,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-27 03:55:40,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-27 03:55:40,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-27 03:55:40,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-27 03:55:40,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-27 03:55:40,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-27 03:55:40,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-27 03:55:40,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-27 03:55:40,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-27 03:55:40,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-27 03:55:40,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-27 03:55:40,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-27 03:55:40,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-27 03:55:40,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-27 03:55:40,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-27 03:55:40,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-27 03:55:40,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-27 03:55:40,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-27 03:55:40,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-27 03:55:40,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-27 03:55:40,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-27 03:55:40,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-27 03:55:40,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-27 03:55:40,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-27 03:55:40,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-27 03:55:40,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-27 03:55:40,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-27 03:55:40,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-27 03:55:40,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-27 03:55:40,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-27 03:55:40,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-27 03:55:40,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-27 03:55:40,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-27 03:55:40,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-27 03:55:40,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-27 03:55:40,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-27 03:55:40,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-27 03:55:40,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-27 03:55:40,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-27 03:55:40,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-27 03:55:40,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-27 03:55:40,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-27 03:55:40,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-27 03:55:40,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-27 03:55:40,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-27 03:55:40,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-27 03:55:40,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-27 03:55:40,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-27 03:55:40,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-27 03:55:40,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-27 03:55:40,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-27 03:55:40,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-27 03:55:40,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-27 03:55:40,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-27 03:55:40,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-27 03:55:40,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-27 03:55:40,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-27 03:55:40,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-27 03:55:40,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-27 03:55:40,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-27 03:55:40,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-27 03:55:40,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-27 03:55:40,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-27 03:55:40,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-27 03:55:40,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-27 03:55:40,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-27 03:55:40,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-27 03:55:40,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-27 03:55:40,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-27 03:55:40,805 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-27 03:55:40,805 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-27 03:55:40,806 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-27 03:55:41,884 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1524 ms to try 5000 different iterations.  Found a solution that moves 4646 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18232385238608856. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03935570391128727, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05630205534958829, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5162222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03935570391128727, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17766666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-27 03:55:41,920 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1737479722:81 , srv55068605:81 , srv630915047:81 , srv635326073:81 , srv1227722619:82 , srv1927983897:82 , srv368477409:82 , srv437286198:82 , srv548488761:82 , srv1094325529:83 , srv1417857541:83 , srv1444218298:83 , srv1538739766:83 , srv1640616532:83 , srv181611347:83 , srv2121842066:83 , srv310511489:83 , srv328238410:83 , srv38720932:83 , srv937241083:83 , srv941585437:83 , srv1010882045:84 , srv1142559099:84 , srv1466536474:84 , srv1539312760:84 , srv1971714216:84 , srv1991713604:84 , srv206989088:84 , srv2098264623:84 , srv392718648:84 , srv406524753:84 , srv506707850:84 , srv680500438:84 , srv833388346:84 , srv955226935:84 , srv1251297955:85 , srv1677269974:85 , srv1677469211:85 , srv1986981316:85 , srv2089377918:85 , srv1100414175:86 , srv1216645669:86 , srv125527346:86 , srv1300332418:86 , srv1463294836:86 , srv1467188500:86 , srv1909649209:86 , srv1941893105:86 , srv29504339:86 , srv343084686:86 , srv37362796:86 , srv426148073:86 , srv579509762:86 , srv616022491:86 , srv681143533:86 , srv73556177:86 , srv923290372:86 , srv1704132265:87 , srv2028969550:87 , srv203808242:87 , srv218430266:87 , srv496108236:87 , srv617399575:87 , srv1221873958:88 , srv1581140820:88 , srv1707891155:88 , srv1754191090:88 , srv1962043673:88 , srv1996656467:88 , srv503762364:88 , srv616172402:88 , srv776493440:88 , srv810111236:88 , srv125591564:89 , srv1443009817:89 , srv1541552339:89 , srv200158938:89 , srv398707149:89 , srv595014969:89 , srv667924310:89 , srv991491216:89 , srv1004668178:90 , srv1089809605:90 , srv153562850:90 , srv1652022331:90 , srv2040347569:90 , srv324488649:90 , srv933450361:90 , srv97231620:90 , srv1755336894:91 , srv237300623:91 , srv378045804:91 , srv1799369018:92 , srv223116909:92 , srv115458547:94 , srv1543442543:94 , srv1462411611:95 , srv1545321396:96 , srv1247106695:98 , srv2034671227:439 }
2023-04-27 03:55:41,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-27 03:55:41,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1581140820=28, srv1754191090=36, srv37362796=66, srv328238410=63, srv398707149=70, srv1216645669=7, srv1909649209=40, srv2034671227=50, srv125527346=12, srv1100414175=4, srv680500438=87, srv1417857541=15, srv1704132265=33, srv153562850=22, srv1677469211=32, srv1010882045=1, srv506707850=76, srv630915047=84, srv616022491=81, srv616172402=82, srv833388346=92, srv1463294836=19, srv810111236=91, srv1247106695=10, srv125591564=13, srv1462411611=18, srv503762364=75, srv1227722619=9, srv1927983897=41, srv955226935=97, srv29504339=60, srv406524753=71, srv1545321396=27, srv310511489=61, srv426148073=72, srv1677269974=31, srv55068605=78, srv115458547=6, srv1089809605=2, srv181611347=39, srv1466536474=20, srv941585437=96, srv1996656467=47, srv1971714216=44, srv937241083=95, srv776493440=90, srv1799369018=38, srv1991713604=46, srv496108236=74, srv1443009817=16, srv200158938=48, srv2040347569=52, srv579509762=79, srv2121842066=56, srv203808242=51, srv206989088=53, srv548488761=77, srv378045804=67, srv667924310=86, srv2089377918=54, srv1962043673=43, srv1538739766=23, srv635326073=85, srv991491216=99, srv1467188500=21, srv923290372=93, srv2098264623=55, srv1652022331=30, srv324488649=62, srv1251297955=11, srv2028969550=49, srv343084686=64, srv392718648=69, srv1986981316=45, srv933450361=94, srv97231620=98, srv218430266=57, srv1300332418=14, srv1541552339=25, srv1444218298=17, srv1941893105=42, srv1640616532=29, srv1004668178=0, srv237300623=59, srv1539312760=24, srv223116909=58, srv1221873958=8, srv1142559099=5, srv437286198=73, srv368477409=65, srv73556177=89, srv1755336894=37, srv617399575=83, srv595014969=80, srv681143533=88, srv1737479722=35, srv1707891155=34, srv38720932=68, srv1094325529=3, srv1543442543=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-27 03:55:41,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-27 03:55:41,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-27 03:55:41,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-27 03:55:41,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-27 03:55:41,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-27 03:55:41,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-27 03:55:41,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-27 03:55:41,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-27 03:55:41,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-27 03:55:41,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-27 03:55:41,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-27 03:55:41,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-27 03:55:41,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-27 03:55:41,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-27 03:55:41,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-27 03:55:41,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-27 03:55:41,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-27 03:55:41,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-27 03:55:41,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-27 03:55:41,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-27 03:55:41,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-27 03:55:41,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-27 03:55:41,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-27 03:55:41,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-27 03:55:41,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-27 03:55:41,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-27 03:55:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-27 03:55:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-27 03:55:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-27 03:55:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-27 03:55:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-27 03:55:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-27 03:55:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-27 03:55:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-27 03:55:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-27 03:55:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-27 03:55:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-27 03:55:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-27 03:55:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-27 03:55:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-27 03:55:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-27 03:55:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-27 03:55:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-27 03:55:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-27 03:55:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-27 03:55:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-27 03:55:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-27 03:55:41,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-27 03:55:41,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-27 03:55:41,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-27 03:55:41,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-27 03:55:41,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-27 03:55:41,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-27 03:55:41,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-27 03:55:41,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-27 03:55:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-27 03:55:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-27 03:55:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-27 03:55:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-27 03:55:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-27 03:55:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-27 03:55:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-27 03:55:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-27 03:55:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-27 03:55:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-27 03:55:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-27 03:55:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-27 03:55:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-27 03:55:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-27 03:55:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-27 03:55:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-27 03:55:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-27 03:55:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-27 03:55:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-27 03:55:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-27 03:55:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-27 03:55:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-27 03:55:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-27 03:55:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-27 03:55:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-27 03:55:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-27 03:55:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-27 03:55:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-27 03:55:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-27 03:55:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-27 03:55:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-27 03:55:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-27 03:55:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-27 03:55:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-27 03:55:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-27 03:55:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-27 03:55:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-27 03:55:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-27 03:55:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-27 03:55:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-27 03:55:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-27 03:55:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-27 03:55:41,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-27 03:55:41,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-27 03:55:41,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-27 03:55:41,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-27 03:55:42,167 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03935570391128727, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05630205534958829, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03935570391128727, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17766666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-27 03:55:42,168 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-27 03:55:42,168 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18229132039876844, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03935570391128727, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05630205534958829, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03935570391128727, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17766666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-27 03:55:42,584 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 594 ms to try 5000 different iterations.  Found a solution that moves 2643 regions; Going from a computed imbalance of 0.18229132039876844 to a new imbalance of 0.016599723878730523. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021833868077743658); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03355481971231444, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2936666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021833868077743658); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-27 03:55:42,596 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1462411611:75 , srv1545321396:75 , srv1247106695:76 , srv115458547:79 , srv1543442543:79 , srv1799369018:79 , srv223116909:80 , srv153562850:81 , srv1541552339:81 , srv1652022331:81 , srv1755336894:81 , srv1004668178:82 , srv1221873958:82 , srv125591564:82 , srv1443009817:82 , srv1962043673:82 , srv378045804:82 , srv503762364:82 , srv810111236:82 , srv933450361:82 , srv97231620:82 , srv991491216:82 , srv1089809605:83 , srv1581140820:83 , srv1996656467:83 , srv2040347569:83 , srv237300623:83 , srv324488649:83 , srv1100414175:84 , srv1463294836:84 , srv1707891155:84 , srv2028969550:84 , srv398707149:84 , srv426148073:84 , srv667924310:84 , srv73556177:84 , srv1300332418:85 , srv1677269974:85 , srv1909649209:85 , srv200158938:85 , srv37362796:85 , srv406524753:85 , srv616172402:85 , srv776493440:85 , srv125527346:86 , srv1704132265:86 , srv218430266:86 , srv29504339:86 , srv343084686:86 , srv496108236:86 , srv595014969:86 , srv617399575:86 , srv681143533:86 , srv1216645669:87 , srv1251297955:87 , srv1444218298:87 , srv1538739766:87 , srv1539312760:87 , srv203808242:87 , srv206989088:87 , srv2121842066:87 , srv310511489:87 , srv579509762:87 , srv616022491:87 , srv680500438:87 , srv1010882045:88 , srv1467188500:88 , srv1754191090:88 , srv181611347:88 , srv1941893105:88 , srv1986981316:88 , srv328238410:88 , srv833388346:88 , srv941585437:88 , srv1417857541:89 , srv1466536474:89 , srv1640616532:89 , srv1677469211:89 , srv1927983897:89 , srv2089377918:89 , srv368477409:89 , srv437286198:89 , srv55068605:89 , srv923290372:89 , srv955226935:89 , srv1094325529:90 , srv1142559099:90 , srv1227722619:90 , srv1971714216:90 , srv1991713604:90 , srv2098264623:90 , srv38720932:90 , srv392718648:90 , srv506707850:90 , srv548488761:90 , srv635326073:90 , srv1737479722:91 , srv630915047:91 , srv937241083:91 , srv2034671227:529 }
2023-04-27 03:55:42,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-27 03:55:42,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1581140820=28, srv1754191090=36, srv37362796=66, srv328238410=63, srv398707149=70, srv1216645669=7, srv1909649209=40, srv2034671227=50, srv125527346=12, srv1100414175=4, srv680500438=87, srv1417857541=15, srv1704132265=33, srv153562850=22, srv1677469211=32, srv1010882045=1, srv506707850=76, srv630915047=84, srv616022491=81, srv616172402=82, srv833388346=92, srv1463294836=19, srv810111236=91, srv1247106695=10, srv125591564=13, srv1462411611=18, srv503762364=75, srv1227722619=9, srv1927983897=41, srv955226935=97, srv29504339=60, srv406524753=71, srv1545321396=27, srv310511489=61, srv426148073=72, srv1677269974=31, srv55068605=78, srv115458547=6, srv1089809605=2, srv181611347=39, srv1466536474=20, srv941585437=96, srv1996656467=47, srv1971714216=44, srv937241083=95, srv776493440=90, srv1799369018=38, srv1991713604=46, srv496108236=74, srv1443009817=16, srv200158938=48, srv2040347569=52, srv579509762=79, srv2121842066=56, srv203808242=51, srv206989088=53, srv548488761=77, srv378045804=67, srv667924310=86, srv2089377918=54, srv1962043673=43, srv1538739766=23, srv635326073=85, srv991491216=99, srv1467188500=21, srv923290372=93, srv2098264623=55, srv1652022331=30, srv324488649=62, srv1251297955=11, srv2028969550=49, srv343084686=64, srv392718648=69, srv1986981316=45, srv933450361=94, srv97231620=98, srv218430266=57, srv1300332418=14, srv1541552339=25, srv1444218298=17, srv1941893105=42, srv1640616532=29, srv1004668178=0, srv237300623=59, srv1539312760=24, srv223116909=58, srv1221873958=8, srv1142559099=5, srv437286198=73, srv368477409=65, srv73556177=89, srv1755336894=37, srv617399575=83, srv595014969=80, srv681143533=88, srv1737479722=35, srv1707891155=34, srv38720932=68, srv1094325529=3, srv1543442543=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-27 03:55:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-27 03:55:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-27 03:55:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-27 03:55:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-27 03:55:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-27 03:55:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-27 03:55:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-27 03:55:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-27 03:55:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-27 03:55:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-27 03:55:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-27 03:55:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-27 03:55:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-27 03:55:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-27 03:55:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-27 03:55:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-27 03:55:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-27 03:55:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-27 03:55:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-27 03:55:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-27 03:55:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-27 03:55:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-27 03:55:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-27 03:55:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-27 03:55:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-27 03:55:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-27 03:55:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-27 03:55:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-27 03:55:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-27 03:55:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-27 03:55:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-27 03:55:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-27 03:55:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-27 03:55:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-27 03:55:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-27 03:55:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-27 03:55:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-27 03:55:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-27 03:55:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-27 03:55:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-27 03:55:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-27 03:55:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-27 03:55:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-27 03:55:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-27 03:55:42,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-27 03:55:42,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-27 03:55:42,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-27 03:55:42,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-27 03:55:42,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-27 03:55:42,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-27 03:55:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-27 03:55:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-27 03:55:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-27 03:55:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-27 03:55:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-27 03:55:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-27 03:55:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-27 03:55:42,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-27 03:55:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-27 03:55:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-27 03:55:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-27 03:55:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-27 03:55:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-27 03:55:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-27 03:55:42,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-27 03:55:42,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-27 03:55:42,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-27 03:55:42,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-27 03:55:42,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-27 03:55:42,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-27 03:55:42,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-27 03:55:42,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-27 03:55:42,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-27 03:55:42,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-27 03:55:42,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-27 03:55:42,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-27 03:55:42,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-27 03:55:42,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-27 03:55:42,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-27 03:55:42,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-27 03:55:42,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-27 03:55:42,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-27 03:55:42,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-27 03:55:42,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-27 03:55:42,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-27 03:55:42,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-27 03:55:42,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-27 03:55:42,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-27 03:55:42,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-27 03:55:42,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-27 03:55:42,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-27 03:55:42,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-27 03:55:42,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-27 03:55:42,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-27 03:55:42,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-27 03:55:42,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-27 03:55:42,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-27 03:55:42,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-27 03:55:42,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-27 03:55:42,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-27 03:55:42,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-27 03:55:42,810 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021833868077743658); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03355481971231444, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021833868077743658); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-27 03:55:42,810 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-27 03:55:42,810 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016581217197179286, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021833868077743658); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03355481971231444, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021833868077743658); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-27 03:55:43,085 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 408 ms to try 5000 different iterations.  Found a solution that moves 1016 regions; Going from a computed imbalance of 0.016581217197179286 to a new imbalance of 0.0013252792494701441. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00698813417189378); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010358338378419612); MoveCostFunction : (multiplier=7.0, imbalance=0.11288888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00698813417189378); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.001); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-27 03:55:43,087 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1754191090:82 , srv1971714216:82 , srv1142559099:83 , srv1247106695:83 , srv1467188500:83 , srv1941893105:83 , srv1991713604:83 , srv203808242:83 , srv2089377918:83 , srv237300623:83 , srv392718648:83 , srv506707850:83 , srv595014969:83 , srv616172402:83 , srv923290372:83 , srv1089809605:84 , srv1094325529:84 , srv115458547:84 , srv1216645669:84 , srv1466536474:84 , srv1543442543:84 , srv1677469211:84 , srv1704132265:84 , srv1986981316:84 , srv200158938:84 , srv2098264623:84 , srv218430266:84 , srv378045804:84 , srv398707149:84 , srv667924310:84 , srv937241083:84 , srv955226935:84 , srv1227722619:85 , srv1251297955:85 , srv1417857541:85 , srv1640616532:85 , srv1707891155:85 , srv1737479722:85 , srv223116909:85 , srv324488649:85 , srv38720932:85 , srv496108236:85 , srv548488761:85 , srv579509762:85 , srv617399575:85 , srv681143533:85 , srv776493440:85 , srv97231620:85 , srv1004668178:86 , srv1010882045:86 , srv125527346:86 , srv153562850:86 , srv1538739766:86 , srv1539312760:86 , srv1545321396:86 , srv1652022331:86 , srv1755336894:86 , srv1799369018:86 , srv181611347:86 , srv1909649209:86 , srv1996656467:86 , srv2028969550:86 , srv2040347569:86 , srv206989088:86 , srv29504339:86 , srv328238410:86 , srv343084686:86 , srv368477409:86 , srv37362796:86 , srv616022491:86 , srv630915047:86 , srv680500438:86 , srv833388346:86 , srv933450361:86 , srv941585437:86 , srv991491216:86 , srv1100414175:87 , srv1221873958:87 , srv125591564:87 , srv1300332418:87 , srv1443009817:87 , srv1444218298:87 , srv1462411611:87 , srv1463294836:87 , srv1541552339:87 , srv1581140820:87 , srv1927983897:87 , srv437286198:87 , srv55068605:87 , srv635326073:87 , srv73556177:87 , srv1677269974:88 , srv1962043673:88 , srv2121842066:88 , srv310511489:88 , srv406524753:88 , srv426148073:88 , srv503762364:88 , srv810111236:88 , srv2034671227:552 }
2023-04-27 03:55:43,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-27 03:55:43,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1581140820=28, srv1754191090=36, srv37362796=66, srv328238410=63, srv398707149=70, srv1216645669=7, srv1909649209=40, srv2034671227=50, srv125527346=12, srv1100414175=4, srv680500438=87, srv1417857541=15, srv1704132265=33, srv153562850=22, srv1677469211=32, srv1010882045=1, srv506707850=76, srv630915047=84, srv616022491=81, srv616172402=82, srv833388346=92, srv1463294836=19, srv810111236=91, srv1247106695=10, srv125591564=13, srv1462411611=18, srv503762364=75, srv1227722619=9, srv1927983897=41, srv955226935=97, srv29504339=60, srv406524753=71, srv1545321396=27, srv310511489=61, srv426148073=72, srv1677269974=31, srv55068605=78, srv115458547=6, srv1089809605=2, srv181611347=39, srv1466536474=20, srv941585437=96, srv1996656467=47, srv1971714216=44, srv937241083=95, srv776493440=90, srv1799369018=38, srv1991713604=46, srv496108236=74, srv1443009817=16, srv200158938=48, srv2040347569=52, srv579509762=79, srv2121842066=56, srv203808242=51, srv206989088=53, srv548488761=77, srv378045804=67, srv667924310=86, srv2089377918=54, srv1962043673=43, srv1538739766=23, srv635326073=85, srv991491216=99, srv1467188500=21, srv923290372=93, srv2098264623=55, srv1652022331=30, srv324488649=62, srv1251297955=11, srv2028969550=49, srv343084686=64, srv392718648=69, srv1986981316=45, srv933450361=94, srv97231620=98, srv218430266=57, srv1300332418=14, srv1541552339=25, srv1444218298=17, srv1941893105=42, srv1640616532=29, srv1004668178=0, srv237300623=59, srv1539312760=24, srv223116909=58, srv1221873958=8, srv1142559099=5, srv437286198=73, srv368477409=65, srv73556177=89, srv1755336894=37, srv617399575=83, srv595014969=80, srv681143533=88, srv1737479722=35, srv1707891155=34, srv38720932=68, srv1094325529=3, srv1543442543=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-27 03:55:43,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-27 03:55:43,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-27 03:55:43,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-27 03:55:43,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-27 03:55:43,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-27 03:55:43,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-27 03:55:43,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-27 03:55:43,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-27 03:55:43,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-27 03:55:43,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-27 03:55:43,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-27 03:55:43,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-27 03:55:43,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-27 03:55:43,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-27 03:55:43,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-27 03:55:43,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-27 03:55:43,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-27 03:55:43,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-27 03:55:43,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-27 03:55:43,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-27 03:55:43,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-27 03:55:43,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-27 03:55:43,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-27 03:55:43,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-27 03:55:43,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-27 03:55:43,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-27 03:55:43,230 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00698813417189378); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010358338378419612); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00698813417189378); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.001); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-27 03:55:43,230 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-27 03:55:43,230 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0013181650654156393, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00698813417189378); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010358338378419612); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00698813417189378); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.001); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-27 03:55:43,661 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 539 ms to try 5000 different iterations.  Found a solution that moves 267 regions; Going from a computed imbalance of 0.0013181650654156393 to a new imbalance of 1.9343247189750356E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.029666666666666668, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-27 03:55:43,662 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2040347569:83 , srv616022491:83 , srv667924310:83 , srv1004668178:84 , srv1010882045:84 , srv125527346:84 , srv1300332418:84 , srv1443009817:84 , srv1462411611:84 , srv1541552339:84 , srv1581140820:84 , srv1652022331:84 , srv1677269974:84 , srv1677469211:84 , srv1754191090:84 , srv1755336894:84 , srv1927983897:84 , srv1941893105:84 , srv1962043673:84 , srv200158938:84 , srv2098264623:84 , srv2121842066:84 , srv29504339:84 , srv310511489:84 , srv324488649:84 , srv343084686:84 , srv426148073:84 , srv437286198:84 , srv496108236:84 , srv503762364:84 , srv55068605:84 , srv617399575:84 , srv630915047:84 , srv635326073:84 , srv776493440:84 , srv833388346:84 , srv923290372:84 , srv933450361:84 , srv937241083:84 , srv97231620:84 , srv1089809605:85 , srv1094325529:85 , srv1142559099:85 , srv115458547:85 , srv1216645669:85 , srv1227722619:85 , srv1247106695:85 , srv1251297955:85 , srv125591564:85 , srv1417857541:85 , srv1444218298:85 , srv1463294836:85 , srv1466536474:85 , srv1467188500:85 , srv153562850:85 , srv1539312760:85 , srv1543442543:85 , srv1545321396:85 , srv1640616532:85 , srv1704132265:85 , srv1707891155:85 , srv1737479722:85 , srv1799369018:85 , srv181611347:85 , srv1909649209:85 , srv1986981316:85 , srv1991713604:85 , srv1996656467:85 , srv2028969550:85 , srv203808242:85 , srv206989088:85 , srv2089377918:85 , srv218430266:85 , srv237300623:85 , srv328238410:85 , srv368477409:85 , srv37362796:85 , srv378045804:85 , srv38720932:85 , srv392718648:85 , srv398707149:85 , srv406524753:85 , srv506707850:85 , srv548488761:85 , srv579509762:85 , srv595014969:85 , srv680500438:85 , srv681143533:85 , srv73556177:85 , srv810111236:85 , srv941585437:85 , srv955226935:85 , srv991491216:85 , srv1100414175:86 , srv1221873958:86 , srv1538739766:86 , srv1971714216:86 , srv223116909:86 , srv616172402:86 , srv2034671227:622 }
2023-04-27 03:55:43,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-27 03:55:43,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1581140820=28, srv1754191090=36, srv37362796=66, srv328238410=63, srv398707149=70, srv1216645669=7, srv1909649209=40, srv2034671227=50, srv125527346=12, srv1100414175=4, srv680500438=87, srv1417857541=15, srv1704132265=33, srv153562850=22, srv1677469211=32, srv1010882045=1, srv506707850=76, srv630915047=84, srv616022491=81, srv616172402=82, srv833388346=92, srv1463294836=19, srv810111236=91, srv1247106695=10, srv125591564=13, srv1462411611=18, srv503762364=75, srv1227722619=9, srv1927983897=41, srv955226935=97, srv29504339=60, srv406524753=71, srv1545321396=27, srv310511489=61, srv426148073=72, srv1677269974=31, srv55068605=78, srv115458547=6, srv1089809605=2, srv181611347=39, srv1466536474=20, srv941585437=96, srv1996656467=47, srv1971714216=44, srv937241083=95, srv776493440=90, srv1799369018=38, srv1991713604=46, srv496108236=74, srv1443009817=16, srv200158938=48, srv2040347569=52, srv579509762=79, srv2121842066=56, srv203808242=51, srv206989088=53, srv548488761=77, srv378045804=67, srv667924310=86, srv2089377918=54, srv1962043673=43, srv1538739766=23, srv635326073=85, srv991491216=99, srv1467188500=21, srv923290372=93, srv2098264623=55, srv1652022331=30, srv324488649=62, srv1251297955=11, srv2028969550=49, srv343084686=64, srv392718648=69, srv1986981316=45, srv933450361=94, srv97231620=98, srv218430266=57, srv1300332418=14, srv1541552339=25, srv1444218298=17, srv1941893105=42, srv1640616532=29, srv1004668178=0, srv237300623=59, srv1539312760=24, srv223116909=58, srv1221873958=8, srv1142559099=5, srv437286198=73, srv368477409=65, srv73556177=89, srv1755336894=37, srv617399575=83, srv595014969=80, srv681143533=88, srv1737479722=35, srv1707891155=34, srv38720932=68, srv1094325529=3, srv1543442543=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-27 03:55:43,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-27 03:55:43,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-27 03:55:43,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-27 03:55:43,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-27 03:55:43,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-27 03:55:43,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-27 03:55:43,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-27 03:55:43,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-27 03:55:43,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-27 03:55:43,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-27 03:55:43,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-27 03:55:43,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-27 03:55:43,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-27 03:55:43,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-27 03:55:43,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-27 03:55:43,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-27 03:55:43,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-27 03:55:43,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-27 03:55:43,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-27 03:55:43,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-27 03:55:43,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-27 03:55:43,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-27 03:55:43,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-27 03:55:43,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-27 03:55:43,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-27 03:55:43,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-27 03:55:43,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-27 03:55:43,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-27 03:55:43,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-27 03:55:43,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-27 03:55:43,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-27 03:55:43,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-27 03:55:43,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-27 03:55:43,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-27 03:55:43,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-27 03:55:43,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-27 03:55:43,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-27 03:55:43,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-27 03:55:43,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-27 03:55:43,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-27 03:55:43,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-27 03:55:43,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-27 03:55:43,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-27 03:55:43,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-27 03:55:43,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-27 03:55:43,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-27 03:55:43,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-27 03:55:43,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-27 03:55:43,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-27 03:55:43,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-27 03:55:43,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-27 03:55:43,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-27 03:55:43,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-27 03:55:43,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-27 03:55:43,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-27 03:55:43,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-27 03:55:43,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-27 03:55:43,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-27 03:55:43,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-27 03:55:43,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-27 03:55:43,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-27 03:55:43,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-27 03:55:43,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-27 03:55:43,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-27 03:55:43,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-27 03:55:43,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-27 03:55:43,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-27 03:55:43,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-27 03:55:43,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-27 03:55:43,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-27 03:55:43,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-27 03:55:43,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-27 03:55:43,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-27 03:55:43,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-27 03:55:43,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-27 03:55:43,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-27 03:55:43,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-27 03:55:43,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-27 03:55:43,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-27 03:55:43,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-27 03:55:43,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-27 03:55:43,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-27 03:55:43,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-27 03:55:43,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-27 03:55:43,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-27 03:55:43,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-27 03:55:43,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-27 03:55:43,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-27 03:55:43,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-27 03:55:43,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-27 03:55:43,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-27 03:55:43,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-27 03:55:43,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-27 03:55:43,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-27 03:55:43,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-27 03:55:43,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-27 03:55:43,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-27 03:55:43,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-27 03:55:43,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-27 03:55:43,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-27 03:55:43,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-27 03:55:43,842 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.747367323054488E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-27 03:55:43,842 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv2040347569:83 , srv616022491:83 , srv667924310:83 , srv1004668178:84 , srv1010882045:84 , srv125527346:84 , srv1300332418:84 , srv1443009817:84 , srv1462411611:84 , srv1541552339:84 , srv1581140820:84 , srv1652022331:84 , srv1677269974:84 , srv1677469211:84 , srv1754191090:84 , srv1755336894:84 , srv1927983897:84 , srv1941893105:84 , srv1962043673:84 , srv200158938:84 , srv2098264623:84 , srv2121842066:84 , srv29504339:84 , srv310511489:84 , srv324488649:84 , srv343084686:84 , srv426148073:84 , srv437286198:84 , srv496108236:84 , srv503762364:84 , srv55068605:84 , srv617399575:84 , srv630915047:84 , srv635326073:84 , srv776493440:84 , srv833388346:84 , srv923290372:84 , srv933450361:84 , srv937241083:84 , srv97231620:84 , srv1089809605:85 , srv1094325529:85 , srv1142559099:85 , srv115458547:85 , srv1216645669:85 , srv1227722619:85 , srv1247106695:85 , srv1251297955:85 , srv125591564:85 , srv1417857541:85 , srv1444218298:85 , srv1463294836:85 , srv1466536474:85 , srv1467188500:85 , srv153562850:85 , srv1539312760:85 , srv1543442543:85 , srv1545321396:85 , srv1640616532:85 , srv1704132265:85 , srv1707891155:85 , srv1737479722:85 , srv1799369018:85 , srv181611347:85 , srv1909649209:85 , srv1986981316:85 , srv1991713604:85 , srv1996656467:85 , srv2028969550:85 , srv203808242:85 , srv206989088:85 , srv2089377918:85 , srv218430266:85 , srv237300623:85 , srv328238410:85 , srv368477409:85 , srv37362796:85 , srv378045804:85 , srv38720932:85 , srv392718648:85 , srv398707149:85 , srv406524753:85 , srv506707850:85 , srv548488761:85 , srv579509762:85 , srv595014969:85 , srv680500438:85 , srv681143533:85 , srv73556177:85 , srv810111236:85 , srv941585437:85 , srv955226935:85 , srv991491216:85 , srv1100414175:86 , srv1221873958:86 , srv1538739766:86 , srv1971714216:86 , srv223116909:86 , srv616172402:86 , srv2034671227:622 }
2023-04-27 03:55:43,880 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=375 (was 329) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=12310 (was 12771)
2023-04-27 03:55:43,900 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=375, ProcessCount=170, AvailableMemoryMB=12309
2023-04-27 03:55:43,901 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-27 03:55:43,901 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-04-27 03:55:43,919 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010323723:84 , srv1026300462:84 , srv1084642183:84 , srv1086933255:84 , srv1121072755:84 , srv1138737066:84 , srv1163955704:84 , srv1164084004:84 , srv1179270416:84 , srv1331890915:84 , srv153120765:84 , srv1537591860:84 , srv1635542651:84 , srv1909346865:84 , srv1920082534:84 , srv1960126899:84 , srv2134249741:84 , srv2138598800:84 , srv299348479:84 , srv336432420:84 , srv345492532:84 , srv4512047:84 , srv708844916:84 , srv748336595:84 , srv845774712:84 , srv882649540:84 , srv920918429:84 , srv9777076:84 , srv983832566:84 , srv1392618527:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-04-27 03:55:43,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-27 03:55:43,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv708844916=23, srv2138598800=18, srv2134249741=17, srv920918429=27, srv983832566=29, srv1010323723=0, srv336432420=20, srv1960126899=16, srv153120765=11, srv1537591860=12, srv1121072755=4, srv1920082534=15, srv748336595=24, srv1026300462=1, srv1331890915=9, srv1909346865=14, srv1084642183=2, srv1086933255=3, srv1392618527=10, srv1138737066=5, srv1635542651=13, srv1163955704=6, srv1179270416=8, srv4512047=22, srv9777076=28, srv299348479=19, srv345492532=21, srv882649540=26, srv1164084004=7, srv845774712=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-27 03:55:43,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-27 03:55:43,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-27 03:55:43,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-27 03:55:43,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-27 03:55:43,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-27 03:55:43,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-27 03:55:43,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-27 03:55:43,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-27 03:55:43,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-27 03:55:43,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-27 03:55:43,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-27 03:55:43,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-27 03:55:43,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-27 03:55:43,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-27 03:55:43,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-27 03:55:43,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-27 03:55:43,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-27 03:55:43,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-27 03:55:43,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-27 03:55:43,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-27 03:55:43,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-27 03:55:43,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-27 03:55:43,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-27 03:55:43,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-27 03:55:43,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-27 03:55:43,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-27 03:55:43,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-27 03:55:43,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-27 03:55:43,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-27 03:55:43,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-27 03:55:43,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-27 03:55:43,967 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1196731157973834, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-27 03:55:43,967 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991257012934447, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1196731157973834, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-04-27 03:57:43,943 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6368672 different iterations.  Found a solution that moves 1844 regions; Going from a computed imbalance of 0.991257012934447 to a new imbalance of 4.303987990979898E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.682962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-27 03:57:43,950 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1010323723:90 , srv1026300462:90 , srv1084642183:90 , srv1086933255:90 , srv1121072755:90 , srv1138737066:90 , srv1163955704:90 , srv1164084004:90 , srv1179270416:90 , srv1331890915:90 , srv1392618527:90 , srv153120765:90 , srv1537591860:90 , srv1635542651:90 , srv1909346865:90 , srv1920082534:90 , srv1960126899:90 , srv2134249741:90 , srv2138598800:90 , srv299348479:90 , srv336432420:90 , srv345492532:90 , srv4512047:90 , srv708844916:90 , srv748336595:90 , srv845774712:90 , srv882649540:90 , srv920918429:90 , srv9777076:90 , srv983832566:90 }
2023-04-27 03:57:43,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-27 03:57:43,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv708844916=23, srv2138598800=18, srv2134249741=17, srv920918429=27, srv983832566=29, srv1010323723=0, srv336432420=20, srv1960126899=16, srv153120765=11, srv1537591860=12, srv1121072755=4, srv1920082534=15, srv748336595=24, srv1026300462=1, srv1331890915=9, srv1909346865=14, srv1084642183=2, srv1086933255=3, srv1392618527=10, srv1138737066=5, srv1635542651=13, srv1163955704=6, srv1179270416=8, srv4512047=22, srv9777076=28, srv299348479=19, srv345492532=21, srv882649540=26, srv1164084004=7, srv845774712=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-27 03:57:43,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-27 03:57:43,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-27 03:57:43,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-27 03:57:43,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-27 03:57:43,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-27 03:57:43,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-27 03:57:43,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-27 03:57:43,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-27 03:57:43,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-27 03:57:43,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-27 03:57:43,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-27 03:57:43,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-27 03:57:43,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-27 03:57:43,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-27 03:57:43,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-27 03:57:43,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-27 03:57:43,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-27 03:57:43,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-27 03:57:43,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-27 03:57:43,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-27 03:57:43,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-27 03:57:43,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-27 03:57:43,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-27 03:57:43,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-27 03:57:43,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-27 03:57:43,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-27 03:57:43,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-27 03:57:43,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-27 03:57:43,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-27 03:57:43,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-27 03:57:43,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-27 03:57:43,971 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-27 03:57:43,971 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1010323723:90 , srv1026300462:90 , srv1084642183:90 , srv1086933255:90 , srv1121072755:90 , srv1138737066:90 , srv1163955704:90 , srv1164084004:90 , srv1179270416:90 , srv1331890915:90 , srv1392618527:90 , srv153120765:90 , srv1537591860:90 , srv1635542651:90 , srv1909346865:90 , srv1920082534:90 , srv1960126899:90 , srv2134249741:90 , srv2138598800:90 , srv299348479:90 , srv336432420:90 , srv345492532:90 , srv4512047:90 , srv708844916:90 , srv748336595:90 , srv845774712:90 , srv882649540:90 , srv920918429:90 , srv9777076:90 , srv983832566:90 }
2023-04-27 03:57:43,993 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=327 (was 375), ProcessCount=170 (was 170), AvailableMemoryMB=12077 (was 12309)