2024-02-19 14:53:16,933 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-02-19 14:53:17,266 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-02-19 14:53:17,389 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-02-19 14:53:17,451 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-19 14:53:17,459 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-02-19 14:53:17,475 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=298, ProcessCount=167, AvailableMemoryMB=10125
2024-02-19 14:53:17,477 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-19 14:53:17,477 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-02-19 14:53:17,614 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1049513033:84 , srv1057226281:84 , srv1125622695:84 , srv1164112054:84 , srv116623037:84 , srv1169778610:84 , srv1171472976:84 , srv1178215884:84 , srv1198519598:84 , srv1273738867:84 , srv1282862470:84 , srv1283442336:84 , srv129047637:84 , srv1298695136:84 , srv1301932341:84 , srv1326214828:84 , srv1342512273:84 , srv1359274223:84 , srv1391910659:84 , srv1392395322:84 , srv1402664725:84 , srv143275040:84 , srv1442062984:84 , srv1496692655:84 , srv149691365:84 , srv1515829391:84 , srv1556628439:84 , srv156814347:84 , srv1585387055:84 , srv1608047768:84 , srv1616713781:84 , srv1632074385:84 , srv1640691196:84 , srv1653846777:84 , srv1667067160:84 , srv1672661927:84 , srv1691395826:84 , srv1695415860:84 , srv1714702323:84 , srv1716316075:84 , srv1767450265:84 , srv1836433469:84 , srv1849896537:84 , srv1862917103:84 , srv188177644:84 , srv1896646165:84 , srv1898211459:84 , srv1899132255:84 , srv1922469793:84 , srv1972074531:84 , srv1989686465:84 , srv1999506633:84 , srv2011639915:84 , srv2017352144:84 , srv2021712011:84 , srv2042308569:84 , srv2058244715:84 , srv2068277942:84 , srv2075860905:84 , srv2078165602:84 , srv2093076102:84 , srv2110375766:84 , srv2146900945:84 , srv23342143:84 , srv23629130:84 , srv237446338:84 , srv283505789:84 , srv296424653:84 , srv307256651:84 , srv359022424:84 , srv365301091:84 , srv389301337:84 , srv42727702:84 , srv46467679:84 , srv482301679:84 , srv493116267:84 , srv500388193:84 , srv504014300:84 , srv505695456:84 , srv542273841:84 , srv558168695:84 , srv57528992:84 , srv578447421:84 , srv605914607:84 , srv609172768:84 , srv629045399:84 , srv74946594:84 , srv750996678:84 , srv758759300:84 , srv760829767:84 , srv796000013:84 , srv840488425:84 , srv874672317:84 , srv879824489:84 , srv902419007:84 , srv904501941:84 , srv905291964:84 , srv924196953:84 , srv983468604:84 , srv596882849:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-02-19 14:53:17,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-19 14:53:17,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv237446338=65, srv1273738867=9, srv904501941=96, srv1632074385=31, srv359022424=69, srv1392395322=19, srv1898211459=46, srv1585387055=28, srv1896646165=45, srv2093076102=60, srv1342512273=16, srv1836433469=41, srv2021712011=54, srv796000013=91, srv1402664725=20, srv129047637=12, srv1169778610=5, srv307256651=68, srv1164112054=3, srv1695415860=37, srv389301337=71, srv188177644=44, srv840488425=92, srv1972074531=49, srv1849896537=42, srv1653846777=33, srv504014300=77, srv283505789=66, srv905291964=97, srv750996678=88, srv1714702323=38, srv874672317=93, srv1198519598=8, srv1672661927=35, srv983468604=99, srv57528992=81, srv542273841=79, srv2017352144=53, srv1691395826=36, srv42727702=72, srv1125622695=2, srv482301679=74, srv1608047768=29, srv629045399=86, srv1359274223=17, srv46467679=73, srv2078165602=59, srv605914607=84, srv1283442336=11, srv1640691196=32, srv1178215884=7, srv609172768=85, srv1767450265=40, srv2146900945=62, srv149691365=24, srv23629130=64, srv879824489=94, srv1862917103=43, srv578447421=82, srv2068277942=57, srv902419007=95, srv1556628439=26, srv500388193=76, srv74946594=87, srv558168695=80, srv156814347=27, srv23342143=63, srv1716316075=39, srv493116267=75, srv1326214828=15, srv2110375766=61, srv1667067160=34, srv2075860905=58, srv296424653=67, srv1049513033=0, srv1301932341=14, srv1496692655=23, srv1922469793=48, srv505695456=78, srv1298695136=13, srv1171472976=6, srv1282862470=10, srv143275040=21, srv2058244715=56, srv1899132255=47, srv1616713781=30, srv1989686465=50, srv2042308569=55, srv2011639915=52, srv596882849=83, srv1515829391=25, srv365301091=70, srv1442062984=22, srv1391910659=18, srv1057226281=1, srv758759300=89, srv1999506633=51, srv924196953=98, srv760829767=90, srv116623037=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-19 14:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-19 14:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-19 14:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-19 14:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-19 14:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-19 14:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-19 14:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-19 14:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-19 14:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-19 14:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-19 14:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-19 14:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-19 14:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-19 14:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-19 14:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-19 14:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-19 14:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-19 14:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-19 14:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-19 14:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-19 14:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-19 14:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-19 14:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-19 14:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-19 14:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-19 14:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-19 14:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-19 14:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-19 14:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-19 14:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-19 14:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-19 14:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-19 14:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-19 14:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-19 14:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-19 14:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-19 14:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-19 14:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-19 14:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-19 14:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-19 14:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-19 14:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-19 14:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-19 14:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-19 14:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-19 14:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-19 14:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-19 14:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-19 14:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-19 14:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-19 14:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-19 14:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-19 14:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-19 14:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-19 14:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-19 14:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-19 14:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-19 14:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-19 14:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-19 14:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-19 14:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-19 14:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-19 14:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-19 14:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-19 14:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-19 14:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-19 14:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-19 14:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-19 14:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-19 14:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-19 14:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-19 14:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-19 14:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-19 14:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-19 14:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-19 14:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-19 14:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-19 14:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-19 14:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-19 14:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-19 14:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-19 14:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-19 14:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-19 14:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-19 14:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-19 14:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-19 14:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-19 14:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-19 14:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-19 14:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-19 14:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-19 14:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-19 14:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-19 14:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-19 14:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-19 14:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-19 14:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-19 14:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-19 14:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-19 14:53:17,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-19 14:53:17,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-19 14:53:17,914 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-19 14:53:17,914 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-19 14:53:17,915 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-19 14:53:18,270 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 501 ms to try 5000 different iterations.  Found a solution that moves 4722 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1737133263999774. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03869069264008718, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.059032605269024714, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5246666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03869069264008718, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.169, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23416666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-19 14:53:18,281 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1342512273:82 , srv1862917103:82 , srv1125622695:83 , srv1198519598:83 , srv389301337:83 , srv46467679:83 , srv609172768:83 , srv874672317:83 , srv1049513033:84 , srv1282862470:84 , srv129047637:84 , srv1496692655:84 , srv1585387055:84 , srv1640691196:84 , srv1898211459:84 , srv2068277942:84 , srv796000013:84 , srv840488425:84 , srv924196953:84 , srv1164112054:85 , srv1298695136:85 , srv1326214828:85 , srv1392395322:85 , srv1849896537:85 , srv2017352144:85 , srv296424653:85 , srv307256651:85 , srv359022424:85 , srv42727702:85 , srv504014300:85 , srv542273841:85 , srv578447421:85 , srv750996678:85 , srv1273738867:86 , srv1301932341:86 , srv1402664725:86 , srv1515829391:86 , srv1667067160:86 , srv1714702323:86 , srv1836433469:86 , srv1896646165:86 , srv2021712011:86 , srv2146900945:86 , srv23342143:86 , srv23629130:86 , srv365301091:86 , srv500388193:86 , srv558168695:86 , srv605914607:86 , srv758759300:86 , srv904501941:86 , srv905291964:86 , srv1057226281:87 , srv149691365:87 , srv1608047768:87 , srv1691395826:87 , srv1716316075:87 , srv188177644:87 , srv1972074531:87 , srv1989686465:87 , srv1999506633:87 , srv2078165602:87 , srv2093076102:87 , srv2110375766:87 , srv760829767:87 , srv879824489:87 , srv902419007:87 , srv983468604:87 , srv1169778610:88 , srv1171472976:88 , srv1178215884:88 , srv1283442336:88 , srv1442062984:88 , srv1616713781:88 , srv1653846777:88 , srv1672661927:88 , srv1767450265:88 , srv1922469793:88 , srv2011639915:88 , srv2042308569:88 , srv2075860905:88 , srv283505789:88 , srv57528992:88 , srv116623037:89 , srv143275040:89 , srv1632074385:89 , srv237446338:89 , srv482301679:89 , srv505695456:89 , srv1359274223:90 , srv1391910659:90 , srv156814347:90 , srv1695415860:90 , srv1899132255:90 , srv2058244715:91 , srv74946594:91 , srv629045399:92 , srv1556628439:93 , srv493116267:93 , srv596882849:434 }
2024-02-19 14:53:18,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-19 14:53:18,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv237446338=65, srv1273738867=9, srv904501941=96, srv1632074385=31, srv359022424=69, srv1392395322=19, srv1898211459=46, srv1585387055=28, srv1896646165=45, srv2093076102=60, srv1342512273=16, srv1836433469=41, srv2021712011=54, srv796000013=91, srv1402664725=20, srv129047637=12, srv1169778610=5, srv307256651=68, srv1164112054=3, srv1695415860=37, srv389301337=71, srv188177644=44, srv840488425=92, srv1972074531=49, srv1849896537=42, srv1653846777=33, srv504014300=77, srv283505789=66, srv905291964=97, srv750996678=88, srv1714702323=38, srv874672317=93, srv1198519598=8, srv1672661927=35, srv983468604=99, srv57528992=81, srv542273841=79, srv2017352144=53, srv1691395826=36, srv42727702=72, srv1125622695=2, srv482301679=74, srv1608047768=29, srv629045399=86, srv1359274223=17, srv46467679=73, srv2078165602=59, srv605914607=84, srv1283442336=11, srv1640691196=32, srv1178215884=7, srv609172768=85, srv1767450265=40, srv2146900945=62, srv149691365=24, srv23629130=64, srv879824489=94, srv1862917103=43, srv578447421=82, srv2068277942=57, srv902419007=95, srv1556628439=26, srv500388193=76, srv74946594=87, srv558168695=80, srv156814347=27, srv23342143=63, srv1716316075=39, srv493116267=75, srv1326214828=15, srv2110375766=61, srv1667067160=34, srv2075860905=58, srv296424653=67, srv1049513033=0, srv1301932341=14, srv1496692655=23, srv1922469793=48, srv505695456=78, srv1298695136=13, srv1171472976=6, srv1282862470=10, srv143275040=21, srv2058244715=56, srv1899132255=47, srv1616713781=30, srv1989686465=50, srv2042308569=55, srv2011639915=52, srv596882849=83, srv1515829391=25, srv365301091=70, srv1442062984=22, srv1391910659=18, srv1057226281=1, srv758759300=89, srv1999506633=51, srv924196953=98, srv760829767=90, srv116623037=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-19 14:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-19 14:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-19 14:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-19 14:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-19 14:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-19 14:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-19 14:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-19 14:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-19 14:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-19 14:53:18,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-19 14:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-19 14:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-19 14:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-19 14:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-19 14:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-19 14:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-19 14:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-19 14:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-19 14:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-19 14:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-19 14:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-19 14:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-19 14:53:18,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-19 14:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-19 14:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-19 14:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-19 14:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-19 14:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-19 14:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-19 14:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-19 14:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-19 14:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-19 14:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-19 14:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-19 14:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-19 14:53:18,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-19 14:53:18,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-19 14:53:18,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-19 14:53:18,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-19 14:53:18,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-19 14:53:18,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-19 14:53:18,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-19 14:53:18,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-19 14:53:18,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-19 14:53:18,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-19 14:53:18,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-19 14:53:18,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-19 14:53:18,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-19 14:53:18,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-19 14:53:18,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-19 14:53:18,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-19 14:53:18,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-19 14:53:18,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-19 14:53:18,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-19 14:53:18,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-19 14:53:18,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-19 14:53:18,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-19 14:53:18,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-19 14:53:18,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-19 14:53:18,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-19 14:53:18,438 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03869069264008718, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.059032605269024714, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03869069264008718, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.169, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23416666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-19 14:53:18,439 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-19 14:53:18,439 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17368026224928318, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03869069264008718, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.059032605269024714, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03869069264008718, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.169, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23416666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-19 14:53:18,781 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 453 ms to try 5000 different iterations.  Found a solution that moves 2616 regions; Going from a computed imbalance of 0.17368026224928318 to a new imbalance of 0.016103854692741768. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020827140867933833); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036181361349331635, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2906666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020827140867933833); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-19 14:53:18,796 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1556628439:79 , srv493116267:80 , srv74946594:80 , srv1171472976:81 , srv1359274223:81 , srv237446338:81 , srv629045399:81 , srv1391910659:82 , srv1442062984:82 , srv2058244715:82 , srv482301679:82 , srv1169778610:83 , srv1178215884:83 , srv1301932341:83 , srv143275040:83 , srv149691365:83 , srv156814347:83 , srv1632074385:83 , srv1653846777:83 , srv1716316075:83 , srv1899132255:83 , srv2042308569:83 , srv2075860905:83 , srv2078165602:83 , srv505695456:83 , srv57528992:83 , srv1057226281:84 , srv1283442336:84 , srv1616713781:84 , srv1691395826:84 , srv1767450265:84 , srv188177644:84 , srv1972074531:84 , srv1999506633:84 , srv2093076102:84 , srv23342143:84 , srv902419007:84 , srv116623037:85 , srv1392395322:85 , srv1608047768:85 , srv1672661927:85 , srv1695415860:85 , srv1836433469:85 , srv1922469793:85 , srv283505789:85 , srv905291964:85 , srv1125622695:86 , srv1164112054:86 , srv1273738867:86 , srv1515829391:86 , srv1714702323:86 , srv2110375766:86 , srv365301091:86 , srv578447421:86 , srv760829767:86 , srv879824489:86 , srv983468604:86 , srv1298695136:87 , srv1402664725:87 , srv1667067160:87 , srv2011639915:87 , srv2021712011:87 , srv2068277942:87 , srv23629130:87 , srv307256651:87 , srv42727702:87 , srv504014300:87 , srv542273841:87 , srv558168695:87 , srv605914607:87 , srv750996678:87 , srv758759300:87 , srv840488425:87 , srv904501941:87 , srv1198519598:88 , srv129047637:88 , srv1326214828:88 , srv1342512273:88 , srv1496692655:88 , srv1898211459:88 , srv1989686465:88 , srv2017352144:88 , srv46467679:88 , srv500388193:88 , srv874672317:88 , srv924196953:88 , srv1049513033:89 , srv1282862470:89 , srv1585387055:89 , srv1849896537:89 , srv1896646165:89 , srv359022424:89 , srv1640691196:90 , srv2146900945:90 , srv1862917103:91 , srv296424653:91 , srv389301337:91 , srv609172768:91 , srv796000013:91 , srv596882849:525 }
2024-02-19 14:53:18,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-19 14:53:18,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv237446338=65, srv1273738867=9, srv904501941=96, srv1632074385=31, srv359022424=69, srv1392395322=19, srv1898211459=46, srv1585387055=28, srv1896646165=45, srv2093076102=60, srv1342512273=16, srv1836433469=41, srv2021712011=54, srv796000013=91, srv1402664725=20, srv129047637=12, srv1169778610=5, srv307256651=68, srv1164112054=3, srv1695415860=37, srv389301337=71, srv188177644=44, srv840488425=92, srv1972074531=49, srv1849896537=42, srv1653846777=33, srv504014300=77, srv283505789=66, srv905291964=97, srv750996678=88, srv1714702323=38, srv874672317=93, srv1198519598=8, srv1672661927=35, srv983468604=99, srv57528992=81, srv542273841=79, srv2017352144=53, srv1691395826=36, srv42727702=72, srv1125622695=2, srv482301679=74, srv1608047768=29, srv629045399=86, srv1359274223=17, srv46467679=73, srv2078165602=59, srv605914607=84, srv1283442336=11, srv1640691196=32, srv1178215884=7, srv609172768=85, srv1767450265=40, srv2146900945=62, srv149691365=24, srv23629130=64, srv879824489=94, srv1862917103=43, srv578447421=82, srv2068277942=57, srv902419007=95, srv1556628439=26, srv500388193=76, srv74946594=87, srv558168695=80, srv156814347=27, srv23342143=63, srv1716316075=39, srv493116267=75, srv1326214828=15, srv2110375766=61, srv1667067160=34, srv2075860905=58, srv296424653=67, srv1049513033=0, srv1301932341=14, srv1496692655=23, srv1922469793=48, srv505695456=78, srv1298695136=13, srv1171472976=6, srv1282862470=10, srv143275040=21, srv2058244715=56, srv1899132255=47, srv1616713781=30, srv1989686465=50, srv2042308569=55, srv2011639915=52, srv596882849=83, srv1515829391=25, srv365301091=70, srv1442062984=22, srv1391910659=18, srv1057226281=1, srv758759300=89, srv1999506633=51, srv924196953=98, srv760829767=90, srv116623037=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-19 14:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-19 14:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-19 14:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-19 14:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-19 14:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-19 14:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-19 14:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-19 14:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-19 14:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-19 14:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-19 14:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-19 14:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-19 14:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-19 14:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-19 14:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-19 14:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-19 14:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-19 14:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-19 14:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-19 14:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-19 14:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-19 14:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-19 14:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-19 14:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-19 14:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-19 14:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-19 14:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-19 14:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-19 14:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-19 14:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-19 14:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-19 14:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-19 14:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-19 14:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-19 14:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-19 14:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-19 14:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-19 14:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-19 14:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-19 14:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-19 14:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-19 14:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-19 14:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-19 14:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-19 14:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-19 14:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-19 14:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-19 14:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-19 14:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-19 14:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-19 14:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-19 14:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-19 14:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-19 14:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-19 14:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-19 14:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-19 14:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-19 14:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-19 14:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-19 14:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-19 14:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-19 14:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-19 14:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-19 14:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-19 14:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-19 14:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-19 14:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-19 14:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-19 14:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-19 14:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-19 14:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-19 14:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-19 14:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-19 14:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-19 14:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-19 14:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-19 14:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-19 14:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-19 14:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-19 14:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-19 14:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-19 14:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-19 14:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-19 14:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-19 14:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-19 14:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-19 14:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-19 14:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-19 14:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-19 14:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-19 14:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-19 14:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-19 14:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-19 14:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-19 14:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-19 14:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-19 14:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-19 14:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-19 14:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-19 14:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-19 14:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-19 14:53:19,018 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020827140867933833); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036181361349331635, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020827140867933833); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-19 14:53:19,018 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-19 14:53:19,018 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01608553706923135, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020827140867933833); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036181361349331635, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020827140867933833); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-19 14:53:19,485 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 619 ms to try 5000 different iterations.  Found a solution that moves 1031 regions; Going from a computed imbalance of 0.01608553706923135 to a new imbalance of 8.420634101413836E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005447890523954445); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011301400844806558); MoveCostFunction : (multiplier=7.0, imbalance=0.11455555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005447890523954445); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-19 14:53:19,491 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1695415860:82 , srv1989686465:82 , srv2146900945:82 , srv296424653:82 , srv796000013:82 , srv1667067160:83 , srv1849896537:83 , srv1896646165:83 , srv2011639915:83 , srv359022424:83 , srv500388193:83 , srv609172768:83 , srv1049513033:84 , srv116623037:84 , srv1282862470:84 , srv1326214828:84 , srv1402664725:84 , srv156814347:84 , srv1862917103:84 , srv2110375766:84 , srv389301337:84 , srv558168695:84 , srv605914607:84 , srv629045399:84 , srv760829767:84 , srv983468604:84 , srv1273738867:85 , srv129047637:85 , srv143275040:85 , srv1496692655:85 , srv1515829391:85 , srv1556628439:85 , srv1585387055:85 , srv1608047768:85 , srv1616713781:85 , srv1640691196:85 , srv1898211459:85 , srv1922469793:85 , srv1999506633:85 , srv2017352144:85 , srv2021712011:85 , srv2058244715:85 , srv283505789:85 , srv307256651:85 , srv365301091:85 , srv493116267:85 , srv505695456:85 , srv904501941:85 , srv1169778610:86 , srv1178215884:86 , srv1298695136:86 , srv1359274223:86 , srv1391910659:86 , srv1442062984:86 , srv1632074385:86 , srv1672661927:86 , srv1691395826:86 , srv1714702323:86 , srv1836433469:86 , srv188177644:86 , srv1899132255:86 , srv2042308569:86 , srv2075860905:86 , srv2093076102:86 , srv23629130:86 , srv42727702:86 , srv46467679:86 , srv504014300:86 , srv542273841:86 , srv57528992:86 , srv74946594:86 , srv750996678:86 , srv758759300:86 , srv879824489:86 , srv902419007:86 , srv924196953:86 , srv1164112054:87 , srv1198519598:87 , srv1283442336:87 , srv1301932341:87 , srv1392395322:87 , srv149691365:87 , srv1653846777:87 , srv1716316075:87 , srv1767450265:87 , srv2068277942:87 , srv2078165602:87 , srv482301679:87 , srv578447421:87 , srv840488425:87 , srv874672317:87 , srv905291964:87 , srv1057226281:88 , srv1125622695:88 , srv1171472976:88 , srv1342512273:88 , srv1972074531:88 , srv23342143:88 , srv237446338:88 , srv596882849:547 }
2024-02-19 14:53:19,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-19 14:53:19,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv237446338=65, srv1273738867=9, srv904501941=96, srv1632074385=31, srv359022424=69, srv1392395322=19, srv1898211459=46, srv1585387055=28, srv1896646165=45, srv2093076102=60, srv1342512273=16, srv1836433469=41, srv2021712011=54, srv796000013=91, srv1402664725=20, srv129047637=12, srv1169778610=5, srv307256651=68, srv1164112054=3, srv1695415860=37, srv389301337=71, srv188177644=44, srv840488425=92, srv1972074531=49, srv1849896537=42, srv1653846777=33, srv504014300=77, srv283505789=66, srv905291964=97, srv750996678=88, srv1714702323=38, srv874672317=93, srv1198519598=8, srv1672661927=35, srv983468604=99, srv57528992=81, srv542273841=79, srv2017352144=53, srv1691395826=36, srv42727702=72, srv1125622695=2, srv482301679=74, srv1608047768=29, srv629045399=86, srv1359274223=17, srv46467679=73, srv2078165602=59, srv605914607=84, srv1283442336=11, srv1640691196=32, srv1178215884=7, srv609172768=85, srv1767450265=40, srv2146900945=62, srv149691365=24, srv23629130=64, srv879824489=94, srv1862917103=43, srv578447421=82, srv2068277942=57, srv902419007=95, srv1556628439=26, srv500388193=76, srv74946594=87, srv558168695=80, srv156814347=27, srv23342143=63, srv1716316075=39, srv493116267=75, srv1326214828=15, srv2110375766=61, srv1667067160=34, srv2075860905=58, srv296424653=67, srv1049513033=0, srv1301932341=14, srv1496692655=23, srv1922469793=48, srv505695456=78, srv1298695136=13, srv1171472976=6, srv1282862470=10, srv143275040=21, srv2058244715=56, srv1899132255=47, srv1616713781=30, srv1989686465=50, srv2042308569=55, srv2011639915=52, srv596882849=83, srv1515829391=25, srv365301091=70, srv1442062984=22, srv1391910659=18, srv1057226281=1, srv758759300=89, srv1999506633=51, srv924196953=98, srv760829767=90, srv116623037=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-19 14:53:19,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-19 14:53:19,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-19 14:53:19,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-19 14:53:19,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-19 14:53:19,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-19 14:53:19,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-19 14:53:19,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-19 14:53:19,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-19 14:53:19,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-19 14:53:19,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-19 14:53:19,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-19 14:53:19,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-19 14:53:19,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-19 14:53:19,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-19 14:53:19,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-19 14:53:19,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-19 14:53:19,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-19 14:53:19,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-19 14:53:19,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-19 14:53:19,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-19 14:53:19,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-19 14:53:19,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-19 14:53:19,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-19 14:53:19,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-19 14:53:19,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-19 14:53:19,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-19 14:53:19,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-19 14:53:19,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-19 14:53:19,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-19 14:53:19,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-19 14:53:19,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-19 14:53:19,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-19 14:53:19,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-19 14:53:19,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-19 14:53:19,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-19 14:53:19,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-19 14:53:19,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-19 14:53:19,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-19 14:53:19,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-19 14:53:19,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-19 14:53:19,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-19 14:53:19,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-19 14:53:19,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-19 14:53:19,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-19 14:53:19,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-19 14:53:19,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-19 14:53:19,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-19 14:53:19,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-19 14:53:19,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-19 14:53:19,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-19 14:53:19,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-19 14:53:19,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-19 14:53:19,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-19 14:53:19,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-19 14:53:19,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-19 14:53:19,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-19 14:53:19,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-19 14:53:19,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-19 14:53:19,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-19 14:53:19,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-19 14:53:19,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-19 14:53:19,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-19 14:53:19,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-19 14:53:19,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-19 14:53:19,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-19 14:53:19,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-19 14:53:19,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-19 14:53:19,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-19 14:53:19,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-19 14:53:19,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-19 14:53:19,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-19 14:53:19,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-19 14:53:19,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-19 14:53:19,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-19 14:53:19,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-19 14:53:19,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-19 14:53:19,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-19 14:53:19,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-19 14:53:19,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-19 14:53:19,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-19 14:53:19,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-19 14:53:19,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-19 14:53:19,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-19 14:53:19,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-19 14:53:19,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-19 14:53:19,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-19 14:53:19,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-19 14:53:19,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-19 14:53:19,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-19 14:53:19,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-19 14:53:19,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-19 14:53:19,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-19 14:53:19,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-19 14:53:19,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-19 14:53:19,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-19 14:53:19,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-19 14:53:19,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-19 14:53:19,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-19 14:53:19,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-19 14:53:19,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-19 14:53:19,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-19 14:53:19,709 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005447890523954445); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011301400844806558); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005447890523954445); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-19 14:53:19,709 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-19 14:53:19,711 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.348441938419797E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005447890523954445); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011301400844806558); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005447890523954445); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-19 14:53:20,035 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 480 ms to try 5000 different iterations.  Found a solution that moves 253 regions; Going from a computed imbalance of 8.348441938419797E-4 to a new imbalance of 1.24366924791577E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.02811111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-19 14:53:20,037 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1057226281:83 , srv1164112054:83 , srv1283442336:83 , srv1672661927:83 , srv1767450265:83 , srv1899132255:83 , srv1972074531:83 , srv23629130:83 , srv542273841:83 , srv758759300:83 , srv879824489:83 , srv1198519598:84 , srv1298695136:84 , srv1326214828:84 , srv1342512273:84 , srv1392395322:84 , srv156814347:84 , srv1632074385:84 , srv1640691196:84 , srv1653846777:84 , srv1691395826:84 , srv1716316075:84 , srv1896646165:84 , srv1922469793:84 , srv2011639915:84 , srv2017352144:84 , srv2021712011:84 , srv2058244715:84 , srv2068277942:84 , srv2146900945:84 , srv23342143:84 , srv237446338:84 , srv283505789:84 , srv296424653:84 , srv389301337:84 , srv482301679:84 , srv493116267:84 , srv504014300:84 , srv558168695:84 , srv578447421:84 , srv750996678:84 , srv796000013:84 , srv840488425:84 , srv904501941:84 , srv905291964:84 , srv924196953:84 , srv1049513033:85 , srv1125622695:85 , srv116623037:85 , srv1171472976:85 , srv1178215884:85 , srv1282862470:85 , srv129047637:85 , srv1359274223:85 , srv1391910659:85 , srv1402664725:85 , srv143275040:85 , srv1442062984:85 , srv1496692655:85 , srv149691365:85 , srv1515829391:85 , srv1556628439:85 , srv1585387055:85 , srv1608047768:85 , srv1616713781:85 , srv1695415860:85 , srv1714702323:85 , srv1836433469:85 , srv1849896537:85 , srv1862917103:85 , srv188177644:85 , srv1898211459:85 , srv1989686465:85 , srv2042308569:85 , srv2075860905:85 , srv2078165602:85 , srv2093076102:85 , srv2110375766:85 , srv307256651:85 , srv365301091:85 , srv42727702:85 , srv46467679:85 , srv500388193:85 , srv505695456:85 , srv57528992:85 , srv605914607:85 , srv609172768:85 , srv629045399:85 , srv74946594:85 , srv760829767:85 , srv874672317:85 , srv902419007:85 , srv983468604:85 , srv1169778610:86 , srv1273738867:86 , srv1301932341:86 , srv1667067160:86 , srv1999506633:86 , srv359022424:86 , srv596882849:636 }
2024-02-19 14:53:20,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-19 14:53:20,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv237446338=65, srv1273738867=9, srv904501941=96, srv1632074385=31, srv359022424=69, srv1392395322=19, srv1898211459=46, srv1585387055=28, srv1896646165=45, srv2093076102=60, srv1342512273=16, srv1836433469=41, srv2021712011=54, srv796000013=91, srv1402664725=20, srv129047637=12, srv1169778610=5, srv307256651=68, srv1164112054=3, srv1695415860=37, srv389301337=71, srv188177644=44, srv840488425=92, srv1972074531=49, srv1849896537=42, srv1653846777=33, srv504014300=77, srv283505789=66, srv905291964=97, srv750996678=88, srv1714702323=38, srv874672317=93, srv1198519598=8, srv1672661927=35, srv983468604=99, srv57528992=81, srv542273841=79, srv2017352144=53, srv1691395826=36, srv42727702=72, srv1125622695=2, srv482301679=74, srv1608047768=29, srv629045399=86, srv1359274223=17, srv46467679=73, srv2078165602=59, srv605914607=84, srv1283442336=11, srv1640691196=32, srv1178215884=7, srv609172768=85, srv1767450265=40, srv2146900945=62, srv149691365=24, srv23629130=64, srv879824489=94, srv1862917103=43, srv578447421=82, srv2068277942=57, srv902419007=95, srv1556628439=26, srv500388193=76, srv74946594=87, srv558168695=80, srv156814347=27, srv23342143=63, srv1716316075=39, srv493116267=75, srv1326214828=15, srv2110375766=61, srv1667067160=34, srv2075860905=58, srv296424653=67, srv1049513033=0, srv1301932341=14, srv1496692655=23, srv1922469793=48, srv505695456=78, srv1298695136=13, srv1171472976=6, srv1282862470=10, srv143275040=21, srv2058244715=56, srv1899132255=47, srv1616713781=30, srv1989686465=50, srv2042308569=55, srv2011639915=52, srv596882849=83, srv1515829391=25, srv365301091=70, srv1442062984=22, srv1391910659=18, srv1057226281=1, srv758759300=89, srv1999506633=51, srv924196953=98, srv760829767=90, srv116623037=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-19 14:53:20,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-19 14:53:20,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-19 14:53:20,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-19 14:53:20,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-19 14:53:20,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-19 14:53:20,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-19 14:53:20,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-19 14:53:20,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-19 14:53:20,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-19 14:53:20,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-19 14:53:20,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-19 14:53:20,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-19 14:53:20,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-19 14:53:20,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-19 14:53:20,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-19 14:53:20,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-19 14:53:20,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-19 14:53:20,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-19 14:53:20,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-19 14:53:20,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-19 14:53:20,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-19 14:53:20,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-19 14:53:20,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-19 14:53:20,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-19 14:53:20,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-19 14:53:20,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-19 14:53:20,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-19 14:53:20,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-19 14:53:20,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-19 14:53:20,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-19 14:53:20,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-19 14:53:20,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-19 14:53:20,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-19 14:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-19 14:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-19 14:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-19 14:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-19 14:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-19 14:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-19 14:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-19 14:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-19 14:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-19 14:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-19 14:53:20,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-19 14:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-19 14:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-19 14:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-19 14:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-19 14:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-19 14:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-19 14:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-19 14:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-19 14:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-19 14:53:20,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-19 14:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-19 14:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-19 14:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-19 14:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-19 14:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-19 14:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-19 14:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-19 14:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-19 14:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-19 14:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-19 14:53:20,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-19 14:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-19 14:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-19 14:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-19 14:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-19 14:53:20,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-19 14:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-19 14:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-19 14:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-19 14:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-19 14:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-19 14:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-19 14:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-19 14:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-19 14:53:20,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-19 14:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-19 14:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-19 14:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-19 14:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-19 14:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-19 14:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-19 14:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-19 14:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-19 14:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-19 14:53:20,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-19 14:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-19 14:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-19 14:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-19 14:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-19 14:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-19 14:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-19 14:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-19 14:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-19 14:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-19 14:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-19 14:53:20,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-19 14:53:20,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-19 14:53:20,219 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.0665148615191465E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-19 14:53:20,219 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1057226281:83 , srv1164112054:83 , srv1283442336:83 , srv1672661927:83 , srv1767450265:83 , srv1899132255:83 , srv1972074531:83 , srv23629130:83 , srv542273841:83 , srv758759300:83 , srv879824489:83 , srv1198519598:84 , srv1298695136:84 , srv1326214828:84 , srv1342512273:84 , srv1392395322:84 , srv156814347:84 , srv1632074385:84 , srv1640691196:84 , srv1653846777:84 , srv1691395826:84 , srv1716316075:84 , srv1896646165:84 , srv1922469793:84 , srv2011639915:84 , srv2017352144:84 , srv2021712011:84 , srv2058244715:84 , srv2068277942:84 , srv2146900945:84 , srv23342143:84 , srv237446338:84 , srv283505789:84 , srv296424653:84 , srv389301337:84 , srv482301679:84 , srv493116267:84 , srv504014300:84 , srv558168695:84 , srv578447421:84 , srv750996678:84 , srv796000013:84 , srv840488425:84 , srv904501941:84 , srv905291964:84 , srv924196953:84 , srv1049513033:85 , srv1125622695:85 , srv116623037:85 , srv1171472976:85 , srv1178215884:85 , srv1282862470:85 , srv129047637:85 , srv1359274223:85 , srv1391910659:85 , srv1402664725:85 , srv143275040:85 , srv1442062984:85 , srv1496692655:85 , srv149691365:85 , srv1515829391:85 , srv1556628439:85 , srv1585387055:85 , srv1608047768:85 , srv1616713781:85 , srv1695415860:85 , srv1714702323:85 , srv1836433469:85 , srv1849896537:85 , srv1862917103:85 , srv188177644:85 , srv1898211459:85 , srv1989686465:85 , srv2042308569:85 , srv2075860905:85 , srv2078165602:85 , srv2093076102:85 , srv2110375766:85 , srv307256651:85 , srv365301091:85 , srv42727702:85 , srv46467679:85 , srv500388193:85 , srv505695456:85 , srv57528992:85 , srv605914607:85 , srv609172768:85 , srv629045399:85 , srv74946594:85 , srv760829767:85 , srv874672317:85 , srv902419007:85 , srv983468604:85 , srv1169778610:86 , srv1273738867:86 , srv1301932341:86 , srv1667067160:86 , srv1999506633:86 , srv359022424:86 , srv596882849:636 }
2024-02-19 14:53:20,259 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=322 (was 298) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=9355 (was 10125)
2024-02-19 14:53:20,269 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=322, ProcessCount=167, AvailableMemoryMB=9355
2024-02-19 14:53:20,270 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-19 14:53:20,270 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-02-19 14:53:20,282 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1016163991:84 , srv1068121470:84 , srv1114310435:84 , srv1139874828:84 , srv1162495683:84 , srv1171837746:84 , srv1186397631:84 , srv1337499976:84 , srv1357682693:84 , srv1477715981:84 , srv1588551829:84 , srv1590310702:84 , srv1612607561:84 , srv1684939255:84 , srv180530229:84 , srv1848231719:84 , srv1933116139:84 , srv1972343551:84 , srv2126850363:84 , srv350420332:84 , srv382242383:84 , srv497283803:84 , srv540327177:84 , srv561939549:84 , srv700755652:84 , srv808768931:84 , srv851113579:84 , srv905456123:84 , srv9367633:84 , srv1936724303:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-02-19 14:53:20,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-19 14:53:20,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv540327177=23, srv1016163991=0, srv1588551829=10, srv1612607561=12, srv700755652=25, srv1114310435=2, srv1848231719=15, srv1162495683=4, srv1933116139=16, srv1139874828=3, srv1936724303=17, srv497283803=22, srv1186397631=6, srv1972343551=18, srv2126850363=19, srv9367633=29, srv808768931=26, srv1337499976=7, srv1590310702=11, srv1357682693=8, srv180530229=14, srv1068121470=1, srv1477715981=9, srv350420332=20, srv851113579=27, srv382242383=21, srv1684939255=13, srv905456123=28, srv561939549=24, srv1171837746=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-19 14:53:20,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-19 14:53:20,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-19 14:53:20,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-19 14:53:20,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-19 14:53:20,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-19 14:53:20,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-19 14:53:20,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-19 14:53:20,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-19 14:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-19 14:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-19 14:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-19 14:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-19 14:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-19 14:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-19 14:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-19 14:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-19 14:53:20,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-02-19 14:53:20,323 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.134383653923981, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-19 14:53:20,324 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912616481769764, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.134383653923981, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-02-19 14:55:20,304 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6073781 different iterations.  Found a solution that moves 1860 regions; Going from a computed imbalance of 0.9912616481769764 to a new imbalance of 4.341332789166274E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6888888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-19 14:55:20,311 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1016163991:90 , srv1068121470:90 , srv1114310435:90 , srv1139874828:90 , srv1162495683:90 , srv1171837746:90 , srv1186397631:90 , srv1337499976:90 , srv1357682693:90 , srv1477715981:90 , srv1588551829:90 , srv1590310702:90 , srv1612607561:90 , srv1684939255:90 , srv180530229:90 , srv1848231719:90 , srv1933116139:90 , srv1936724303:90 , srv1972343551:90 , srv2126850363:90 , srv350420332:90 , srv382242383:90 , srv497283803:90 , srv540327177:90 , srv561939549:90 , srv700755652:90 , srv808768931:90 , srv851113579:90 , srv905456123:90 , srv9367633:90 }
2024-02-19 14:55:20,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-19 14:55:20,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv540327177=23, srv1016163991=0, srv1588551829=10, srv1612607561=12, srv700755652=25, srv1114310435=2, srv1848231719=15, srv1162495683=4, srv1933116139=16, srv1139874828=3, srv1936724303=17, srv497283803=22, srv1186397631=6, srv1972343551=18, srv2126850363=19, srv9367633=29, srv808768931=26, srv1337499976=7, srv1590310702=11, srv1357682693=8, srv180530229=14, srv1068121470=1, srv1477715981=9, srv350420332=20, srv851113579=27, srv382242383=21, srv1684939255=13, srv905456123=28, srv561939549=24, srv1171837746=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-19 14:55:20,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-19 14:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-19 14:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-19 14:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-19 14:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-19 14:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-19 14:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-19 14:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-19 14:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-19 14:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-19 14:55:20,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-19 14:55:20,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-02-19 14:55:20,329 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-19 14:55:20,330 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1016163991:90 , srv1068121470:90 , srv1114310435:90 , srv1139874828:90 , srv1162495683:90 , srv1171837746:90 , srv1186397631:90 , srv1337499976:90 , srv1357682693:90 , srv1477715981:90 , srv1588551829:90 , srv1590310702:90 , srv1612607561:90 , srv1684939255:90 , srv180530229:90 , srv1848231719:90 , srv1933116139:90 , srv1936724303:90 , srv1972343551:90 , srv2126850363:90 , srv350420332:90 , srv382242383:90 , srv497283803:90 , srv540327177:90 , srv561939549:90 , srv700755652:90 , srv808768931:90 , srv851113579:90 , srv905456123:90 , srv9367633:90 }
2024-02-19 14:55:20,343 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=181 (was 322), ProcessCount=167 (was 167), AvailableMemoryMB=8831 (was 9355)