2023-04-29 15:53:31,911 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-04-29 15:53:32,255 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-04-29 15:53:32,369 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-04-29 15:53:32,441 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-29 15:53:32,449 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-29 15:53:32,463 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=322, ProcessCount=170, AvailableMemoryMB=11406
2023-04-29 15:53:32,466 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-29 15:53:32,466 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-29 15:53:32,609 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1042766780:84 , srv1043299852:84 , srv1079197117:84 , srv1096528599:84 , srv1140738607:84 , srv1152630070:84 , srv1154523174:84 , srv1181076251:84 , srv1226439683:84 , srv1244829863:84 , srv1273400686:84 , srv1281886461:84 , srv1290707221:84 , srv1299401747:84 , srv1355704206:84 , srv135741663:84 , srv1362686613:84 , srv1365676215:84 , srv1368735953:84 , srv1370518971:84 , srv1396936630:84 , srv140492906:84 , srv1405573556:84 , srv1414666043:84 , srv1458151935:84 , srv1470668876:84 , srv1488734954:84 , srv1489288359:84 , srv1489601620:84 , srv1515309736:84 , srv1516627478:84 , srv1532320674:84 , srv1543614199:84 , srv1579429276:84 , srv1582009726:84 , srv1621644150:84 , srv1637546603:84 , srv1693168866:84 , srv17170179:84 , srv1769943098:84 , srv1833064881:84 , srv1835411837:84 , srv1864015678:84 , srv1869252277:84 , srv1909652891:84 , srv1915258825:84 , srv1985297855:84 , srv1987435829:84 , srv1997775450:84 , srv2005995474:84 , srv2006442686:84 , srv2012119259:84 , srv2031871725:84 , srv2065636426:84 , srv2071930834:84 , srv2072362114:84 , srv2086588461:84 , srv2103070054:84 , srv2138451373:84 , srv2146262162:84 , srv222291384:84 , srv242273687:84 , srv243107399:84 , srv246600673:84 , srv250775651:84 , srv270683734:84 , srv287595423:84 , srv287843241:84 , srv30992558:84 , srv323956200:84 , srv354414053:84 , srv379698061:84 , srv387180315:84 , srv420924341:84 , srv428005782:84 , srv458636676:84 , srv465832956:84 , srv474086510:84 , srv484344055:84 , srv484921486:84 , srv51956622:84 , srv546572285:84 , srv577523003:84 , srv579640215:84 , srv618652194:84 , srv620253588:84 , srv648023932:84 , srv650609709:84 , srv655982496:84 , srv660913886:84 , srv677581243:84 , srv747805764:84 , srv786865673:84 , srv815783996:84 , srv823186158:84 , srv856071733:84 , srv857752040:84 , srv880494067:84 , srv898602555:84 , srv64582985:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-04-29 15:53:32,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 15:53:32,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv270683734=65, srv64582985=86, srv1140738607=4, srv898602555=99, srv1368735953=18, srv1470668876=25, srv1405573556=22, srv1543614199=32, srv1997775450=48, srv135741663=15, srv420924341=73, srv1396936630=20, srv17170179=38, srv648023932=87, srv856071733=96, srv1579429276=33, srv243107399=62, srv1043299852=1, srv246600673=63, srv458636676=75, srv857752040=97, srv660913886=90, srv323956200=69, srv1637546603=36, srv465832956=76, srv1864015678=42, srv1915258825=45, srv1869252277=43, srv650609709=88, srv387180315=72, srv1987435829=47, srv2138451373=58, srv618652194=84, srv1244829863=9, srv1281886461=11, srv1290707221=12, srv30992558=68, srv379698061=71, srv51956622=80, srv1833064881=40, srv140492906=21, srv1152630070=5, srv1362686613=16, srv2005995474=49, srv2072362114=55, srv484344055=78, srv2146262162=59, srv1370518971=19, srv2065636426=53, srv1515309736=29, srv2071930834=54, srv546572285=81, srv2012119259=51, srv1365676215=17, srv1835411837=41, srv786865673=93, srv880494067=98, srv242273687=61, srv579640215=83, srv747805764=92, srv1181076251=7, srv1985297855=46, srv1226439683=8, srv2031871725=52, srv1909652891=44, srv484921486=79, srv1769943098=39, srv354414053=70, srv1458151935=24, srv655982496=89, srv222291384=60, srv428005782=74, srv1489288359=27, srv1079197117=2, srv1355704206=14, srv1488734954=26, srv2103070054=57, srv2006442686=50, srv1299401747=13, srv1273400686=10, srv1154523174=6, srv1516627478=30, srv474086510=77, srv1489601620=28, srv1042766780=0, srv1693168866=37, srv1582009726=34, srv2086588461=56, srv1414666043=23, srv287595423=66, srv677581243=91, srv1096528599=3, srv287843241=67, srv620253588=85, srv815783996=94, srv1532320674=31, srv577523003=82, srv250775651=64, srv1621644150=35, srv823186158=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 15:53:32,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 15:53:32,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 15:53:32,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 15:53:32,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 15:53:32,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 15:53:32,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 15:53:32,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 15:53:32,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 15:53:32,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 15:53:32,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 15:53:32,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 15:53:32,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 15:53:32,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 15:53:32,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 15:53:32,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 15:53:32,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 15:53:32,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 15:53:32,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 15:53:32,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 15:53:32,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 15:53:32,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 15:53:32,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 15:53:32,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 15:53:32,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 15:53:32,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 15:53:32,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 15:53:32,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 15:53:32,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 15:53:32,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 15:53:32,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 15:53:32,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-29 15:53:32,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-29 15:53:32,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-29 15:53:32,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-29 15:53:32,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-29 15:53:32,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-29 15:53:32,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-29 15:53:32,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-29 15:53:32,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-29 15:53:32,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-29 15:53:32,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-29 15:53:32,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-29 15:53:32,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-29 15:53:32,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-29 15:53:32,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-29 15:53:32,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-29 15:53:32,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-29 15:53:32,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-29 15:53:32,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-29 15:53:32,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-29 15:53:32,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-29 15:53:32,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-29 15:53:32,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-29 15:53:32,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-29 15:53:32,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-29 15:53:32,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-29 15:53:32,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-29 15:53:32,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-29 15:53:32,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-29 15:53:32,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-29 15:53:32,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-29 15:53:32,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-29 15:53:32,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-29 15:53:32,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-29 15:53:32,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-29 15:53:32,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-29 15:53:32,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-29 15:53:32,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-29 15:53:32,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-29 15:53:32,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-29 15:53:32,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-29 15:53:32,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-29 15:53:32,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-29 15:53:32,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-29 15:53:32,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-29 15:53:32,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-29 15:53:32,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-29 15:53:32,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-29 15:53:32,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-29 15:53:32,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-29 15:53:32,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-29 15:53:32,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-29 15:53:32,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-29 15:53:32,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-29 15:53:32,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-29 15:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-29 15:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-29 15:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-29 15:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-29 15:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-29 15:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-29 15:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-29 15:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-29 15:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-29 15:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-29 15:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-29 15:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-29 15:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-29 15:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-29 15:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-29 15:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-29 15:53:33,132 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 15:53:33,132 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-29 15:53:33,133 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-29 15:53:33,881 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1127 ms to try 5000 different iterations.  Found a solution that moves 4715 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17565262369312998. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03736921742923212, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056280123496538575, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5238888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03736921742923212, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17091666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 15:53:33,895 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1181076251:82 , srv1833064881:82 , srv2138451373:82 , srv222291384:82 , srv650609709:82 , srv1152630070:83 , srv1987435829:83 , srv243107399:83 , srv1096528599:84 , srv1489288359:84 , srv1516627478:84 , srv1543614199:84 , srv1621644150:84 , srv242273687:84 , srv270683734:84 , srv387180315:84 , srv856071733:84 , srv1042766780:85 , srv1290707221:85 , srv1362686613:85 , srv1414666043:85 , srv1458151935:85 , srv1489601620:85 , srv1579429276:85 , srv1582009726:85 , srv1693168866:85 , srv1915258825:85 , srv1985297855:85 , srv1997775450:85 , srv287843241:85 , srv30992558:85 , srv474086510:85 , srv484921486:85 , srv648023932:85 , srv655982496:85 , srv677581243:85 , srv815783996:85 , srv1244829863:86 , srv135741663:86 , srv1515309736:86 , srv17170179:86 , srv246600673:86 , srv250775651:86 , srv323956200:86 , srv420924341:86 , srv428005782:86 , srv579640215:86 , srv1043299852:87 , srv1140738607:87 , srv1226439683:87 , srv1299401747:87 , srv1355704206:87 , srv1370518971:87 , srv1405573556:87 , srv1864015678:87 , srv1909652891:87 , srv2012119259:87 , srv2031871725:87 , srv2065636426:87 , srv2086588461:87 , srv2103070054:87 , srv2146262162:87 , srv287595423:87 , srv379698061:87 , srv51956622:87 , srv660913886:87 , srv857752040:87 , srv880494067:87 , srv1079197117:88 , srv1365676215:88 , srv1368735953:88 , srv1396936630:88 , srv1488734954:88 , srv1532320674:88 , srv1769943098:88 , srv2005995474:88 , srv2006442686:88 , srv2072362114:88 , srv465832956:88 , srv618652194:88 , srv620253588:88 , srv823186158:88 , srv1281886461:89 , srv1869252277:89 , srv484344055:89 , srv577523003:89 , srv747805764:89 , srv140492906:90 , srv354414053:90 , srv458636676:90 , srv898602555:90 , srv1273400686:91 , srv1637546603:91 , srv1154523174:92 , srv1470668876:92 , srv2071930834:92 , srv546572285:93 , srv786865673:93 , srv1835411837:95 , srv64582985:422 }
2023-04-29 15:53:33,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 15:53:33,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv270683734=65, srv64582985=86, srv1140738607=4, srv898602555=99, srv1368735953=18, srv1470668876=25, srv1405573556=22, srv1543614199=32, srv1997775450=48, srv135741663=15, srv420924341=73, srv1396936630=20, srv17170179=38, srv648023932=87, srv856071733=96, srv1579429276=33, srv243107399=62, srv1043299852=1, srv246600673=63, srv458636676=75, srv857752040=97, srv660913886=90, srv323956200=69, srv1637546603=36, srv465832956=76, srv1864015678=42, srv1915258825=45, srv1869252277=43, srv650609709=88, srv387180315=72, srv1987435829=47, srv2138451373=58, srv618652194=84, srv1244829863=9, srv1281886461=11, srv1290707221=12, srv30992558=68, srv379698061=71, srv51956622=80, srv1833064881=40, srv140492906=21, srv1152630070=5, srv1362686613=16, srv2005995474=49, srv2072362114=55, srv484344055=78, srv2146262162=59, srv1370518971=19, srv2065636426=53, srv1515309736=29, srv2071930834=54, srv546572285=81, srv2012119259=51, srv1365676215=17, srv1835411837=41, srv786865673=93, srv880494067=98, srv242273687=61, srv579640215=83, srv747805764=92, srv1181076251=7, srv1985297855=46, srv1226439683=8, srv2031871725=52, srv1909652891=44, srv484921486=79, srv1769943098=39, srv354414053=70, srv1458151935=24, srv655982496=89, srv222291384=60, srv428005782=74, srv1489288359=27, srv1079197117=2, srv1355704206=14, srv1488734954=26, srv2103070054=57, srv2006442686=50, srv1299401747=13, srv1273400686=10, srv1154523174=6, srv1516627478=30, srv474086510=77, srv1489601620=28, srv1042766780=0, srv1693168866=37, srv1582009726=34, srv2086588461=56, srv1414666043=23, srv287595423=66, srv677581243=91, srv1096528599=3, srv287843241=67, srv620253588=85, srv815783996=94, srv1532320674=31, srv577523003=82, srv250775651=64, srv1621644150=35, srv823186158=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 15:53:33,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 15:53:33,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 15:53:33,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 15:53:33,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 15:53:33,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 15:53:33,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 15:53:33,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 15:53:33,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 15:53:33,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 15:53:33,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 15:53:33,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 15:53:33,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-29 15:53:33,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-29 15:53:33,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-29 15:53:33,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-29 15:53:33,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-29 15:53:33,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-29 15:53:33,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-29 15:53:33,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-29 15:53:33,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-29 15:53:33,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-29 15:53:33,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-29 15:53:33,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-29 15:53:33,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-29 15:53:33,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-29 15:53:33,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-29 15:53:33,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-29 15:53:33,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-29 15:53:33,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-29 15:53:33,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-29 15:53:33,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-29 15:53:33,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-29 15:53:33,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-29 15:53:33,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-29 15:53:33,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-29 15:53:33,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-29 15:53:33,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-29 15:53:33,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-29 15:53:33,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-29 15:53:33,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-29 15:53:33,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-29 15:53:33,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-29 15:53:33,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-29 15:53:33,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-29 15:53:33,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-29 15:53:33,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-29 15:53:34,092 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03736921742923212, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056280123496538575, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03736921742923212, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17091666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 15:53:34,093 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-29 15:53:34,093 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17561960855748332, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03736921742923212, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056280123496538575, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03736921742923212, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17091666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-29 15:53:34,415 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 461 ms to try 5000 different iterations.  Found a solution that moves 2695 regions; Going from a computed imbalance of 0.17561960855748332 to a new imbalance of 0.013508364358294394. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019733225144039676); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0339637358595216, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29944444444444446, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019733225144039676); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01075); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.03958333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 15:53:34,425 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1835411837:78 , srv546572285:78 , srv1273400686:79 , srv1470668876:79 , srv1154523174:80 , srv2071930834:80 , srv458636676:80 , srv1488734954:81 , srv1637546603:81 , srv354414053:81 , srv786865673:81 , srv2006442686:82 , srv2031871725:82 , srv577523003:82 , srv898602555:82 , srv1370518971:83 , srv140492906:83 , srv246600673:83 , srv484344055:83 , srv618652194:83 , srv823186158:83 , srv1079197117:84 , srv1365676215:84 , srv2072362114:84 , srv620253588:84 , srv880494067:84 , srv1140738607:85 , srv1226439683:85 , srv1244829863:85 , srv1290707221:85 , srv1368735953:85 , srv1396936630:85 , srv1489288359:85 , srv1515309736:85 , srv17170179:85 , srv1915258825:85 , srv1997775450:85 , srv2005995474:85 , srv2012119259:85 , srv379698061:85 , srv474086510:85 , srv579640215:85 , srv747805764:85 , srv1281886461:86 , srv1355704206:86 , srv135741663:86 , srv1362686613:86 , srv1458151935:86 , srv1579429276:86 , srv1909652891:86 , srv2065636426:86 , srv2103070054:86 , srv250775651:86 , srv270683734:86 , srv30992558:86 , srv323956200:86 , srv428005782:86 , srv856071733:86 , srv857752040:86 , srv1042766780:87 , srv1299401747:87 , srv1516627478:87 , srv1532320674:87 , srv1543614199:87 , srv1582009726:87 , srv1621644150:87 , srv1769943098:87 , srv1864015678:87 , srv2146262162:87 , srv287843241:87 , srv387180315:87 , srv465832956:87 , srv51956622:87 , srv655982496:87 , srv1043299852:88 , srv1096528599:88 , srv1152630070:88 , srv1985297855:88 , srv1987435829:88 , srv2086588461:88 , srv242273687:88 , srv287595423:88 , srv420924341:88 , srv660913886:88 , srv677581243:88 , srv1405573556:89 , srv1414666043:89 , srv1693168866:89 , srv1869252277:89 , srv243107399:89 , srv484921486:89 , srv815783996:89 , srv1181076251:90 , srv1489601620:90 , srv1833064881:90 , srv648023932:90 , srv650609709:90 , srv2138451373:91 , srv222291384:91 , srv64582985:527 }
2023-04-29 15:53:34,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 15:53:34,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv270683734=65, srv64582985=86, srv1140738607=4, srv898602555=99, srv1368735953=18, srv1470668876=25, srv1405573556=22, srv1543614199=32, srv1997775450=48, srv135741663=15, srv420924341=73, srv1396936630=20, srv17170179=38, srv648023932=87, srv856071733=96, srv1579429276=33, srv243107399=62, srv1043299852=1, srv246600673=63, srv458636676=75, srv857752040=97, srv660913886=90, srv323956200=69, srv1637546603=36, srv465832956=76, srv1864015678=42, srv1915258825=45, srv1869252277=43, srv650609709=88, srv387180315=72, srv1987435829=47, srv2138451373=58, srv618652194=84, srv1244829863=9, srv1281886461=11, srv1290707221=12, srv30992558=68, srv379698061=71, srv51956622=80, srv1833064881=40, srv140492906=21, srv1152630070=5, srv1362686613=16, srv2005995474=49, srv2072362114=55, srv484344055=78, srv2146262162=59, srv1370518971=19, srv2065636426=53, srv1515309736=29, srv2071930834=54, srv546572285=81, srv2012119259=51, srv1365676215=17, srv1835411837=41, srv786865673=93, srv880494067=98, srv242273687=61, srv579640215=83, srv747805764=92, srv1181076251=7, srv1985297855=46, srv1226439683=8, srv2031871725=52, srv1909652891=44, srv484921486=79, srv1769943098=39, srv354414053=70, srv1458151935=24, srv655982496=89, srv222291384=60, srv428005782=74, srv1489288359=27, srv1079197117=2, srv1355704206=14, srv1488734954=26, srv2103070054=57, srv2006442686=50, srv1299401747=13, srv1273400686=10, srv1154523174=6, srv1516627478=30, srv474086510=77, srv1489601620=28, srv1042766780=0, srv1693168866=37, srv1582009726=34, srv2086588461=56, srv1414666043=23, srv287595423=66, srv677581243=91, srv1096528599=3, srv287843241=67, srv620253588=85, srv815783996=94, srv1532320674=31, srv577523003=82, srv250775651=64, srv1621644150=35, srv823186158=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 15:53:34,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 15:53:34,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 15:53:34,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 15:53:34,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 15:53:34,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 15:53:34,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 15:53:34,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 15:53:34,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-29 15:53:34,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-29 15:53:34,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-29 15:53:34,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-29 15:53:34,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-29 15:53:34,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-29 15:53:34,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-29 15:53:34,625 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019733225144039676); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0339637358595216, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019733225144039676); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01075); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.03958333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 15:53:34,626 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-29 15:53:34,626 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01348949356496084, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019733225144039676); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0339637358595216, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019733225144039676); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01075); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.03958333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-29 15:53:34,882 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 392 ms to try 5000 different iterations.  Found a solution that moves 988 regions; Going from a computed imbalance of 0.01348949356496084 to a new imbalance of 8.860465005216531E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.003770442472503547); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006267511942419625); MoveCostFunction : (multiplier=7.0, imbalance=0.10977777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.003770442472503547); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 15:53:34,885 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1869252277:80 , srv1043299852:82 , srv1281886461:82 , srv1405573556:82 , srv1489601620:82 , srv287595423:82 , srv1532320674:83 , srv1769943098:83 , srv465832956:83 , srv484921486:83 , srv648023932:83 , srv660913886:83 , srv1299401747:84 , srv1355704206:84 , srv1414666043:84 , srv1864015678:84 , srv1985297855:84 , srv2086588461:84 , srv2103070054:84 , srv2146262162:84 , srv420924341:84 , srv51956622:84 , srv677581243:84 , srv786865673:84 , srv815783996:84 , srv1042766780:85 , srv1079197117:85 , srv1140738607:85 , srv1181076251:85 , srv135741663:85 , srv1365676215:85 , srv1368735953:85 , srv1396936630:85 , srv140492906:85 , srv1582009726:85 , srv1637546603:85 , srv1693168866:85 , srv1833064881:85 , srv1835411837:85 , srv1909652891:85 , srv2005995474:85 , srv2012119259:85 , srv2065636426:85 , srv2138451373:85 , srv222291384:85 , srv243107399:85 , srv379698061:85 , srv484344055:85 , srv577523003:85 , srv1096528599:86 , srv1154523174:86 , srv1226439683:86 , srv1515309736:86 , srv1516627478:86 , srv1621644150:86 , srv17170179:86 , srv1987435829:86 , srv2071930834:86 , srv2072362114:86 , srv242273687:86 , srv287843241:86 , srv323956200:86 , srv354414053:86 , srv428005782:86 , srv579640215:86 , srv618652194:86 , srv650609709:86 , srv655982496:86 , srv747805764:86 , srv857752040:86 , srv898602555:86 , srv1152630070:87 , srv1244829863:87 , srv1290707221:87 , srv1362686613:87 , srv1370518971:87 , srv1458151935:87 , srv1470668876:87 , srv1543614199:87 , srv1579429276:87 , srv1997775450:87 , srv250775651:87 , srv30992558:87 , srv387180315:87 , srv458636676:87 , srv546572285:87 , srv620253588:87 , srv823186158:87 , srv880494067:87 , srv1273400686:88 , srv1488734954:88 , srv1489288359:88 , srv1915258825:88 , srv2006442686:88 , srv2031871725:88 , srv246600673:88 , srv270683734:88 , srv474086510:88 , srv856071733:88 , srv64582985:542 }
2023-04-29 15:53:34,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 15:53:34,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv270683734=65, srv64582985=86, srv1140738607=4, srv898602555=99, srv1368735953=18, srv1470668876=25, srv1405573556=22, srv1543614199=32, srv1997775450=48, srv135741663=15, srv420924341=73, srv1396936630=20, srv17170179=38, srv648023932=87, srv856071733=96, srv1579429276=33, srv243107399=62, srv1043299852=1, srv246600673=63, srv458636676=75, srv857752040=97, srv660913886=90, srv323956200=69, srv1637546603=36, srv465832956=76, srv1864015678=42, srv1915258825=45, srv1869252277=43, srv650609709=88, srv387180315=72, srv1987435829=47, srv2138451373=58, srv618652194=84, srv1244829863=9, srv1281886461=11, srv1290707221=12, srv30992558=68, srv379698061=71, srv51956622=80, srv1833064881=40, srv140492906=21, srv1152630070=5, srv1362686613=16, srv2005995474=49, srv2072362114=55, srv484344055=78, srv2146262162=59, srv1370518971=19, srv2065636426=53, srv1515309736=29, srv2071930834=54, srv546572285=81, srv2012119259=51, srv1365676215=17, srv1835411837=41, srv786865673=93, srv880494067=98, srv242273687=61, srv579640215=83, srv747805764=92, srv1181076251=7, srv1985297855=46, srv1226439683=8, srv2031871725=52, srv1909652891=44, srv484921486=79, srv1769943098=39, srv354414053=70, srv1458151935=24, srv655982496=89, srv222291384=60, srv428005782=74, srv1489288359=27, srv1079197117=2, srv1355704206=14, srv1488734954=26, srv2103070054=57, srv2006442686=50, srv1299401747=13, srv1273400686=10, srv1154523174=6, srv1516627478=30, srv474086510=77, srv1489601620=28, srv1042766780=0, srv1693168866=37, srv1582009726=34, srv2086588461=56, srv1414666043=23, srv287595423=66, srv677581243=91, srv1096528599=3, srv287843241=67, srv620253588=85, srv815783996=94, srv1532320674=31, srv577523003=82, srv250775651=64, srv1621644150=35, srv823186158=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 15:53:34,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 15:53:34,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 15:53:34,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 15:53:34,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 15:53:34,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 15:53:34,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 15:53:34,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 15:53:34,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 15:53:34,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 15:53:34,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 15:53:34,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 15:53:34,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 15:53:34,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 15:53:34,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 15:53:34,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 15:53:34,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 15:53:34,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 15:53:34,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 15:53:34,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 15:53:34,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 15:53:34,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 15:53:34,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 15:53:34,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 15:53:34,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 15:53:34,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 15:53:34,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 15:53:34,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 15:53:34,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 15:53:34,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 15:53:34,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 15:53:34,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-29 15:53:34,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-29 15:53:34,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-29 15:53:34,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-29 15:53:34,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-29 15:53:34,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-29 15:53:34,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-29 15:53:34,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-29 15:53:34,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-29 15:53:34,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-29 15:53:34,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-29 15:53:34,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-29 15:53:34,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-29 15:53:34,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-29 15:53:34,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-29 15:53:34,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-29 15:53:34,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-29 15:53:34,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-29 15:53:34,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-29 15:53:34,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-29 15:53:34,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-29 15:53:34,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-29 15:53:34,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-29 15:53:34,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-29 15:53:34,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-29 15:53:34,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-29 15:53:34,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-29 15:53:34,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-29 15:53:34,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-29 15:53:34,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-29 15:53:34,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-29 15:53:34,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-29 15:53:34,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-29 15:53:34,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-29 15:53:34,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-29 15:53:34,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-29 15:53:34,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-29 15:53:34,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-29 15:53:34,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-29 15:53:34,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-29 15:53:34,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-29 15:53:34,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-29 15:53:34,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-29 15:53:34,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-29 15:53:34,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-29 15:53:34,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-29 15:53:34,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-29 15:53:34,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-29 15:53:34,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-29 15:53:34,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-29 15:53:34,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-29 15:53:34,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-29 15:53:34,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-29 15:53:34,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-29 15:53:34,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-29 15:53:34,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-29 15:53:34,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-29 15:53:34,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-29 15:53:34,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-29 15:53:34,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-29 15:53:34,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-29 15:53:34,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-29 15:53:34,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-29 15:53:34,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-29 15:53:34,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-29 15:53:34,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-29 15:53:34,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-29 15:53:34,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-29 15:53:34,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-29 15:53:34,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-29 15:53:34,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-29 15:53:35,082 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.003770442472503547); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006267511942419625); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.003770442472503547); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 15:53:35,082 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-29 15:53:35,082 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.791283766576268E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.003770442472503547); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006267511942419625); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.003770442472503547); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-29 15:53:35,430 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 490 ms to try 5000 different iterations.  Found a solution that moves 214 regions; Going from a computed imbalance of 8.791283766576268E-4 to a new imbalance of 1.3809529801716964E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.023777777777777776); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 15:53:35,431 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1693168866:83 , srv250775651:83 , srv323956200:83 , srv620253588:83 , srv747805764:83 , srv815783996:83 , srv857752040:83 , srv1096528599:84 , srv1152630070:84 , srv1226439683:84 , srv1244829863:84 , srv1273400686:84 , srv1299401747:84 , srv1355704206:84 , srv1362686613:84 , srv1368735953:84 , srv140492906:84 , srv1405573556:84 , srv1414666043:84 , srv1458151935:84 , srv1470668876:84 , srv1488734954:84 , srv1516627478:84 , srv1543614199:84 , srv1579429276:84 , srv1582009726:84 , srv1769943098:84 , srv1835411837:84 , srv1864015678:84 , srv1869252277:84 , srv1909652891:84 , srv1915258825:84 , srv2005995474:84 , srv2006442686:84 , srv2065636426:84 , srv2071930834:84 , srv2072362114:84 , srv2086588461:84 , srv2138451373:84 , srv2146262162:84 , srv222291384:84 , srv242273687:84 , srv270683734:84 , srv287843241:84 , srv30992558:84 , srv387180315:84 , srv420924341:84 , srv428005782:84 , srv474086510:84 , srv51956622:84 , srv546572285:84 , srv648023932:84 , srv650609709:84 , srv655982496:84 , srv660913886:84 , srv786865673:84 , srv823186158:84 , srv856071733:84 , srv880494067:84 , srv898602555:84 , srv1042766780:85 , srv1043299852:85 , srv1079197117:85 , srv1140738607:85 , srv1154523174:85 , srv1181076251:85 , srv1281886461:85 , srv1290707221:85 , srv135741663:85 , srv1365676215:85 , srv1370518971:85 , srv1396936630:85 , srv1489288359:85 , srv1489601620:85 , srv1515309736:85 , srv1532320674:85 , srv1621644150:85 , srv1637546603:85 , srv17170179:85 , srv1833064881:85 , srv1985297855:85 , srv1987435829:85 , srv1997775450:85 , srv2012119259:85 , srv2031871725:85 , srv2103070054:85 , srv243107399:85 , srv246600673:85 , srv287595423:85 , srv354414053:85 , srv379698061:85 , srv458636676:85 , srv465832956:85 , srv484344055:85 , srv484921486:85 , srv579640215:85 , srv618652194:85 , srv677581243:85 , srv577523003:86 , srv64582985:651 }
2023-04-29 15:53:35,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 15:53:35,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv270683734=65, srv64582985=86, srv1140738607=4, srv898602555=99, srv1368735953=18, srv1470668876=25, srv1405573556=22, srv1543614199=32, srv1997775450=48, srv135741663=15, srv420924341=73, srv1396936630=20, srv17170179=38, srv648023932=87, srv856071733=96, srv1579429276=33, srv243107399=62, srv1043299852=1, srv246600673=63, srv458636676=75, srv857752040=97, srv660913886=90, srv323956200=69, srv1637546603=36, srv465832956=76, srv1864015678=42, srv1915258825=45, srv1869252277=43, srv650609709=88, srv387180315=72, srv1987435829=47, srv2138451373=58, srv618652194=84, srv1244829863=9, srv1281886461=11, srv1290707221=12, srv30992558=68, srv379698061=71, srv51956622=80, srv1833064881=40, srv140492906=21, srv1152630070=5, srv1362686613=16, srv2005995474=49, srv2072362114=55, srv484344055=78, srv2146262162=59, srv1370518971=19, srv2065636426=53, srv1515309736=29, srv2071930834=54, srv546572285=81, srv2012119259=51, srv1365676215=17, srv1835411837=41, srv786865673=93, srv880494067=98, srv242273687=61, srv579640215=83, srv747805764=92, srv1181076251=7, srv1985297855=46, srv1226439683=8, srv2031871725=52, srv1909652891=44, srv484921486=79, srv1769943098=39, srv354414053=70, srv1458151935=24, srv655982496=89, srv222291384=60, srv428005782=74, srv1489288359=27, srv1079197117=2, srv1355704206=14, srv1488734954=26, srv2103070054=57, srv2006442686=50, srv1299401747=13, srv1273400686=10, srv1154523174=6, srv1516627478=30, srv474086510=77, srv1489601620=28, srv1042766780=0, srv1693168866=37, srv1582009726=34, srv2086588461=56, srv1414666043=23, srv287595423=66, srv677581243=91, srv1096528599=3, srv287843241=67, srv620253588=85, srv815783996=94, srv1532320674=31, srv577523003=82, srv250775651=64, srv1621644150=35, srv823186158=95} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 15:53:35,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 15:53:35,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 15:53:35,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 15:53:35,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 15:53:35,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 15:53:35,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 15:53:35,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 15:53:35,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 15:53:35,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 15:53:35,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 15:53:35,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 15:53:35,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 15:53:35,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 15:53:35,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 15:53:35,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 15:53:35,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 15:53:35,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 15:53:35,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 15:53:35,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 15:53:35,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 15:53:35,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 15:53:35,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 15:53:35,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 15:53:35,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-29 15:53:35,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-29 15:53:35,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-29 15:53:35,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-29 15:53:35,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-29 15:53:35,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-29 15:53:35,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-29 15:53:35,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-29 15:53:35,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-29 15:53:35,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-29 15:53:35,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-29 15:53:35,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-29 15:53:35,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-29 15:53:35,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-29 15:53:35,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-29 15:53:35,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-29 15:53:35,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-29 15:53:35,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-29 15:53:35,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-29 15:53:35,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-29 15:53:35,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-29 15:53:35,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-29 15:53:35,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-29 15:53:35,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-29 15:53:35,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-29 15:53:35,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-29 15:53:35,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-29 15:53:35,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-29 15:53:35,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-29 15:53:35,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-29 15:53:35,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-29 15:53:35,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-29 15:53:35,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-29 15:53:35,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-29 15:53:35,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-29 15:53:35,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-29 15:53:35,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-29 15:53:35,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-29 15:53:35,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-29 15:53:35,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-29 15:53:35,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-29 15:53:35,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-29 15:53:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-29 15:53:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-29 15:53:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-29 15:53:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-29 15:53:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-29 15:53:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-29 15:53:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-29 15:53:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-29 15:53:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-29 15:53:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-29 15:53:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-29 15:53:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-29 15:53:35,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-29 15:53:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-29 15:53:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-29 15:53:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-29 15:53:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-29 15:53:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-29 15:53:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-29 15:53:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-29 15:53:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-29 15:53:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-29 15:53:35,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-29 15:53:35,631 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.2311069774488605E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 15:53:35,631 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1693168866:83 , srv250775651:83 , srv323956200:83 , srv620253588:83 , srv747805764:83 , srv815783996:83 , srv857752040:83 , srv1096528599:84 , srv1152630070:84 , srv1226439683:84 , srv1244829863:84 , srv1273400686:84 , srv1299401747:84 , srv1355704206:84 , srv1362686613:84 , srv1368735953:84 , srv140492906:84 , srv1405573556:84 , srv1414666043:84 , srv1458151935:84 , srv1470668876:84 , srv1488734954:84 , srv1516627478:84 , srv1543614199:84 , srv1579429276:84 , srv1582009726:84 , srv1769943098:84 , srv1835411837:84 , srv1864015678:84 , srv1869252277:84 , srv1909652891:84 , srv1915258825:84 , srv2005995474:84 , srv2006442686:84 , srv2065636426:84 , srv2071930834:84 , srv2072362114:84 , srv2086588461:84 , srv2138451373:84 , srv2146262162:84 , srv222291384:84 , srv242273687:84 , srv270683734:84 , srv287843241:84 , srv30992558:84 , srv387180315:84 , srv420924341:84 , srv428005782:84 , srv474086510:84 , srv51956622:84 , srv546572285:84 , srv648023932:84 , srv650609709:84 , srv655982496:84 , srv660913886:84 , srv786865673:84 , srv823186158:84 , srv856071733:84 , srv880494067:84 , srv898602555:84 , srv1042766780:85 , srv1043299852:85 , srv1079197117:85 , srv1140738607:85 , srv1154523174:85 , srv1181076251:85 , srv1281886461:85 , srv1290707221:85 , srv135741663:85 , srv1365676215:85 , srv1370518971:85 , srv1396936630:85 , srv1489288359:85 , srv1489601620:85 , srv1515309736:85 , srv1532320674:85 , srv1621644150:85 , srv1637546603:85 , srv17170179:85 , srv1833064881:85 , srv1985297855:85 , srv1987435829:85 , srv1997775450:85 , srv2012119259:85 , srv2031871725:85 , srv2103070054:85 , srv243107399:85 , srv246600673:85 , srv287595423:85 , srv354414053:85 , srv379698061:85 , srv458636676:85 , srv465832956:85 , srv484344055:85 , srv484921486:85 , srv579640215:85 , srv618652194:85 , srv677581243:85 , srv577523003:86 , srv64582985:651 }
2023-04-29 15:53:35,657 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=344 (was 322) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=10549 (was 11406)
2023-04-29 15:53:35,673 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=344, ProcessCount=170, AvailableMemoryMB=10547
2023-04-29 15:53:35,674 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-29 15:53:35,674 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-04-29 15:53:35,686 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1056227012:84 , srv1062721474:84 , srv1108825420:84 , srv1144771242:84 , srv1162432832:84 , srv1216277535:84 , srv1246247665:84 , srv1287224849:84 , srv1408165193:84 , srv1437982930:84 , srv1509150881:84 , srv1522682740:84 , srv1559548849:84 , srv1570056138:84 , srv1588800003:84 , srv1755222577:84 , srv1771185088:84 , srv1822010098:84 , srv1882985343:84 , srv2056375325:84 , srv2062842567:84 , srv2080756080:84 , srv2103700403:84 , srv226411273:84 , srv515881744:84 , srv663886110:84 , srv677978041:84 , srv757371318:84 , srv953570239:84 , srv1447034242:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-04-29 15:53:35,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 15:53:35,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1882985343=19, srv515881744=25, srv757371318=28, srv1062721474=1, srv1447034242=10, srv1559548849=13, srv953570239=29, srv1570056138=14, srv1056227012=0, srv1287224849=7, srv1588800003=15, srv1162432832=4, srv1144771242=3, srv1771185088=17, srv1437982930=9, srv1246247665=6, srv1408165193=8, srv2103700403=23, srv2080756080=22, srv1216277535=5, srv1822010098=18, srv226411273=24, srv1755222577=16, srv2062842567=21, srv677978041=27, srv1108825420=2, srv1522682740=12, srv1509150881=11, srv2056375325=20, srv663886110=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 15:53:35,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 15:53:35,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 15:53:35,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 15:53:35,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 15:53:35,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 15:53:35,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 15:53:35,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 15:53:35,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 15:53:35,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 15:53:35,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 15:53:35,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 15:53:35,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 15:53:35,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 15:53:35,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 15:53:35,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 15:53:35,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 15:53:35,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 15:53:35,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-29 15:53:35,726 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1402331784792061, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 15:53:35,727 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912634913430634, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1402331784792061, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-04-29 15:55:35,706 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6950610 different iterations.  Found a solution that moves 1843 regions; Going from a computed imbalance of 0.9912634913430634 to a new imbalance of 4.3016539410932495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6825925925925926, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 15:55:35,712 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1056227012:90 , srv1062721474:90 , srv1108825420:90 , srv1144771242:90 , srv1162432832:90 , srv1216277535:90 , srv1246247665:90 , srv1287224849:90 , srv1408165193:90 , srv1437982930:90 , srv1447034242:90 , srv1509150881:90 , srv1522682740:90 , srv1559548849:90 , srv1570056138:90 , srv1588800003:90 , srv1755222577:90 , srv1771185088:90 , srv1822010098:90 , srv1882985343:90 , srv2056375325:90 , srv2062842567:90 , srv2080756080:90 , srv2103700403:90 , srv226411273:90 , srv515881744:90 , srv663886110:90 , srv677978041:90 , srv757371318:90 , srv953570239:90 }
2023-04-29 15:55:35,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-29 15:55:35,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1882985343=19, srv515881744=25, srv757371318=28, srv1062721474=1, srv1447034242=10, srv1559548849=13, srv953570239=29, srv1570056138=14, srv1056227012=0, srv1287224849=7, srv1588800003=15, srv1162432832=4, srv1144771242=3, srv1771185088=17, srv1437982930=9, srv1246247665=6, srv1408165193=8, srv2103700403=23, srv2080756080=22, srv1216277535=5, srv1822010098=18, srv226411273=24, srv1755222577=16, srv2062842567=21, srv677978041=27, srv1108825420=2, srv1522682740=12, srv1509150881=11, srv2056375325=20, srv663886110=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-29 15:55:35,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-29 15:55:35,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-29 15:55:35,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-29 15:55:35,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-29 15:55:35,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-29 15:55:35,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-29 15:55:35,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-29 15:55:35,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-29 15:55:35,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-29 15:55:35,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-29 15:55:35,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-29 15:55:35,729 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-29 15:55:35,729 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1056227012:90 , srv1062721474:90 , srv1108825420:90 , srv1144771242:90 , srv1162432832:90 , srv1216277535:90 , srv1246247665:90 , srv1287224849:90 , srv1408165193:90 , srv1437982930:90 , srv1447034242:90 , srv1509150881:90 , srv1522682740:90 , srv1559548849:90 , srv1570056138:90 , srv1588800003:90 , srv1755222577:90 , srv1771185088:90 , srv1822010098:90 , srv1882985343:90 , srv2056375325:90 , srv2062842567:90 , srv2080756080:90 , srv2103700403:90 , srv226411273:90 , srv515881744:90 , srv663886110:90 , srv677978041:90 , srv757371318:90 , srv953570239:90 }
2023-04-29 15:55:35,745 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=202 (was 344), ProcessCount=170 (was 170), AvailableMemoryMB=10404 (was 10547)