2023-02-17 20:53:59,372 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-02-17 20:53:59,717 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-17 20:53:59,855 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-17 20:53:59,926 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-17 20:53:59,936 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-17 20:53:59,953 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=369, ProcessCount=171, AvailableMemoryMB=8336 2023-02-17 20:53:59,957 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-17 20:53:59,958 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-17 20:54:00,089 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1032499291:84 , srv1102565795:84 , srv1111852006:84 , srv1125047991:84 , srv1157743095:84 , srv1166009101:84 , srv118078519:84 , srv1183011824:84 , srv1185870863:84 , srv1232701018:84 , srv1257983984:84 , srv1285813645:84 , srv1286258689:84 , srv1295118860:84 , srv130386475:84 , srv1323692036:84 , srv1335568754:84 , srv1339653493:84 , srv140873209:84 , srv1417725961:84 , srv1436725875:84 , srv1471026708:84 , srv1500252463:84 , srv1505164895:84 , srv1505391986:84 , srv1544079401:84 , srv157512818:84 , srv15822157:84 , srv1582972710:84 , srv1601606785:84 , srv1645066627:84 , srv170430479:84 , srv1708420065:84 , srv1724418413:84 , srv1734317169:84 , srv1749230566:84 , srv1752471825:84 , srv1773002901:84 , srv1783706883:84 , srv1792280444:84 , srv1814025136:84 , srv1866059938:84 , srv1868786521:84 , srv1897683111:84 , srv1902836750:84 , srv1904935497:84 , srv1908752373:84 , srv1909345318:84 , srv1910062989:84 , srv1940019092:84 , srv1959040317:84 , srv2055205135:84 , srv2067396665:84 , srv2106052951:84 , srv2127966744:84 , srv220021143:84 , srv221535383:84 , srv23039918:84 , srv238082475:84 , srv245127297:84 , srv255179497:84 , srv263531332:84 , srv299910445:84 , srv320357861:84 , srv328943074:84 , srv343628724:84 , srv350352895:84 , srv364739795:84 , srv367777460:84 , srv375847328:84 , srv395822664:84 , srv414083758:84 , srv422348656:84 , srv426228459:84 , srv429867095:84 , srv43000455:84 , srv434284031:84 , srv443280134:84 , srv454672209:84 , srv473667958:84 , srv47662640:84 , srv477001456:84 , srv604763457:84 , srv646400019:84 , srv650764615:84 , srv65957914:84 , srv671355397:84 , srv74199630:84 , srv746802922:84 , srv754737452:84 , srv831788024:84 , srv86167544:84 , srv874381431:84 , srv878595174:84 , srv903511414:84 , srv90362807:84 , srv906908507:84 , srv978780598:84 , srv997565287:84 , srv1493186142:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-02-17 20:54:00,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 20:54:00,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2127966744=55, srv1102565795=1, srv454672209=79, srv422348656=73, srv903511414=95, srv245127297=60, srv15822157=28, srv1959040317=51, srv90362807=96, srv1582972710=29, srv1417725961=19, srv2106052951=54, srv343628724=66, srv1285813645=11, srv1335568754=16, srv1910062989=49, srv263531332=62, srv375847328=70, srv746802922=89, srv874381431=93, srv1902836750=45, srv157512818=27, srv1749230566=36, srv1493186142=22, srv1792280444=40, srv1908752373=47, srv238082475=59, srv367777460=69, srv1814025136=41, srv1295118860=13, srv1866059938=42, srv350352895=67, srv1724418413=34, srv414083758=72, srv434284031=77, srv426228459=74, srv650764615=85, srv1157743095=4, srv1232701018=9, srv395822664=71, srv473667958=80, srv1183011824=7, srv1645066627=31, srv130386475=14, srv1166009101=5, srv1868786521=43, srv1783706883=39, srv1752471825=37, srv118078519=6, srv1323692036=15, srv646400019=84, srv1032499291=0, srv65957914=86, srv671355397=87, srv906908507=97, srv86167544=92, srv1940019092=50, srv299910445=63, srv74199630=88, srv221535383=57, srv429867095=75, srv140873209=18, srv1505391986=25, srv2067396665=53, srv604763457=83, srv1544079401=26, srv1257983984=10, srv1904935497=46, srv1471026708=21, srv2055205135=52, srv220021143=56, srv443280134=78, srv255179497=61, srv878595174=94, srv1773002901=38, srv328943074=65, srv1505164895=24, srv320357861=64, srv754737452=90, srv1500252463=23, srv1125047991=3, srv1601606785=30, srv364739795=68, srv170430479=32, srv47662640=81, srv1286258689=12, srv43000455=76, srv477001456=82, srv1339653493=17, srv23039918=58, srv997565287=99, srv1897683111=44, srv1909345318=48, srv1436725875=20, srv1708420065=33, srv1734317169=35, srv831788024=91, srv978780598=98, srv1185870863=8, srv1111852006=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 20:54:00,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 20:54:00,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 20:54:00,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 20:54:00,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 20:54:00,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 20:54:00,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 20:54:00,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 20:54:00,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 20:54:00,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 20:54:00,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 20:54:00,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 20:54:00,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 20:54:00,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 20:54:00,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 20:54:00,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 20:54:00,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 20:54:00,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 20:54:00,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 20:54:00,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 20:54:00,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 20:54:00,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 20:54:00,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 20:54:00,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 20:54:00,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 20:54:00,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 20:54:00,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 20:54:00,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 20:54:00,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 20:54:00,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 20:54:00,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 20:54:00,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-17 20:54:00,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-17 20:54:00,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-17 20:54:00,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-17 20:54:00,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-17 20:54:00,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-17 20:54:00,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-17 20:54:00,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-17 20:54:00,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-17 20:54:00,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-17 20:54:00,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-17 20:54:00,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-17 20:54:00,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-17 20:54:00,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-17 20:54:00,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-17 20:54:00,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-17 20:54:00,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-17 20:54:00,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-17 20:54:00,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-17 20:54:00,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-17 20:54:00,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-17 20:54:00,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-17 20:54:00,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-17 20:54:00,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-17 20:54:00,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-17 20:54:00,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-17 20:54:00,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-17 20:54:00,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-17 20:54:00,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-17 20:54:00,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-17 20:54:00,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-17 20:54:00,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-17 20:54:00,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-17 20:54:00,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-17 20:54:00,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-17 20:54:00,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-17 20:54:00,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-17 20:54:00,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-17 20:54:00,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-17 20:54:00,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-17 20:54:00,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-17 20:54:00,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-17 20:54:00,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-17 20:54:00,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-17 20:54:00,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-17 20:54:00,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-17 20:54:00,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-17 20:54:00,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-17 20:54:00,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-17 20:54:00,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-17 20:54:00,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-17 20:54:00,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-17 20:54:00,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-17 20:54:00,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-17 20:54:00,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-17 20:54:00,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-17 20:54:00,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-17 20:54:00,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-17 20:54:00,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-17 20:54:00,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-17 20:54:00,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-17 20:54:00,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-17 20:54:00,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-17 20:54:00,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-17 20:54:00,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-17 20:54:00,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-17 20:54:00,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-17 20:54:00,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-17 20:54:00,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-17 20:54:00,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-17 20:54:00,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-17 20:54:00,427 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:54:00,428 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-17 20:54:00,428 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-17 20:54:00,837 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 615 ms to try 5000 different iterations. Found a solution that moves 4682 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1833608679976141. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03891982095154872, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05875628294603155, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5202222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03891982095154872, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1795, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23633333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:54:00,874 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1125047991:83 , srv1335568754:83 , srv1752471825:83 , srv1868786521:83 , srv320357861:83 , srv350352895:83 , srv604763457:83 , srv65957914:83 , srv1111852006:84 , srv1157743095:84 , srv1166009101:84 , srv1285813645:84 , srv1286258689:84 , srv130386475:84 , srv1323692036:84 , srv1417725961:84 , srv1436725875:84 , srv1471026708:84 , srv1505164895:84 , srv157512818:84 , srv2055205135:84 , srv2127966744:84 , srv43000455:84 , srv646400019:84 , srv1339653493:85 , srv1734317169:85 , srv221535383:85 , srv238082475:85 , srv263531332:85 , srv299910445:85 , srv328943074:85 , srv364739795:85 , srv367777460:85 , srv429867095:85 , srv650764615:85 , srv978780598:85 , srv1232701018:86 , srv1544079401:86 , srv1582972710:86 , srv1724418413:86 , srv1749230566:86 , srv1773002901:86 , srv1866059938:86 , srv1908752373:86 , srv1959040317:86 , srv23039918:86 , srv343628724:86 , srv414083758:86 , srv422348656:86 , srv426228459:86 , srv477001456:86 , srv86167544:86 , srv903511414:86 , srv997565287:86 , srv118078519:87 , srv1183011824:87 , srv1185870863:87 , srv1257983984:87 , srv1708420065:87 , srv1783706883:87 , srv1792280444:87 , srv2106052951:87 , srv245127297:87 , srv443280134:87 , srv473667958:87 , srv746802922:87 , srv754737452:87 , srv874381431:87 , srv878595174:87 , srv1102565795:88 , srv1500252463:88 , srv1505391986:88 , srv1645066627:88 , srv1940019092:88 , srv434284031:88 , srv454672209:88 , srv831788024:88 , srv906908507:88 , srv1032499291:89 , srv15822157:89 , srv1904935497:89 , srv1909345318:89 , srv1910062989:89 , srv395822664:89 , srv47662640:89 , srv671355397:89 , srv140873209:90 , srv1814025136:90 , srv1897683111:90 , srv2067396665:90 , srv255179497:90 , srv375847328:90 , srv74199630:90 , srv170430479:91 , srv1902836750:91 , srv220021143:91 , srv90362807:91 , srv1601606785:92 , srv1295118860:93 , srv1493186142:436 } 2023-02-17 20:54:00,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 20:54:00,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2127966744=55, srv1102565795=1, srv454672209=79, srv422348656=73, srv903511414=95, srv245127297=60, srv15822157=28, srv1959040317=51, srv90362807=96, srv1582972710=29, srv1417725961=19, srv2106052951=54, srv343628724=66, srv1285813645=11, srv1335568754=16, srv1910062989=49, srv263531332=62, srv375847328=70, srv746802922=89, srv874381431=93, srv1902836750=45, srv157512818=27, srv1749230566=36, srv1493186142=22, srv1792280444=40, srv1908752373=47, srv238082475=59, srv367777460=69, srv1814025136=41, srv1295118860=13, srv1866059938=42, srv350352895=67, srv1724418413=34, srv414083758=72, srv434284031=77, srv426228459=74, srv650764615=85, srv1157743095=4, srv1232701018=9, srv395822664=71, srv473667958=80, srv1183011824=7, srv1645066627=31, srv130386475=14, srv1166009101=5, srv1868786521=43, srv1783706883=39, srv1752471825=37, srv118078519=6, srv1323692036=15, srv646400019=84, srv1032499291=0, srv65957914=86, srv671355397=87, srv906908507=97, srv86167544=92, srv1940019092=50, srv299910445=63, srv74199630=88, srv221535383=57, srv429867095=75, srv140873209=18, srv1505391986=25, srv2067396665=53, srv604763457=83, srv1544079401=26, srv1257983984=10, srv1904935497=46, srv1471026708=21, srv2055205135=52, srv220021143=56, srv443280134=78, srv255179497=61, srv878595174=94, srv1773002901=38, srv328943074=65, srv1505164895=24, srv320357861=64, srv754737452=90, srv1500252463=23, srv1125047991=3, srv1601606785=30, srv364739795=68, srv170430479=32, srv47662640=81, srv1286258689=12, srv43000455=76, srv477001456=82, srv1339653493=17, srv23039918=58, srv997565287=99, srv1897683111=44, srv1909345318=48, srv1436725875=20, srv1708420065=33, srv1734317169=35, srv831788024=91, srv978780598=98, srv1185870863=8, srv1111852006=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 20:54:00,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 20:54:00,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 20:54:00,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 20:54:00,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 20:54:00,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 20:54:00,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 20:54:00,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 20:54:00,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 20:54:00,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 20:54:00,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 20:54:00,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 20:54:00,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 20:54:00,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 20:54:00,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 20:54:00,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 20:54:00,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 20:54:00,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-17 20:54:00,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-17 20:54:00,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-17 20:54:00,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-17 20:54:00,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-17 20:54:00,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-17 20:54:00,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-17 20:54:00,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-17 20:54:00,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-17 20:54:00,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-17 20:54:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-17 20:54:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-17 20:54:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-17 20:54:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-17 20:54:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-17 20:54:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-17 20:54:01,017 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03891982095154872, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05875628294603155, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03891982095154872, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1795, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23633333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:54:01,018 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-17 20:54:01,018 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18332808393290626, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03891982095154872, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05875628294603155, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03891982095154872, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1795, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23633333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-17 20:54:01,350 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 438 ms to try 5000 different iterations. Found a solution that moves 2619 regions; Going from a computed imbalance of 0.18332808393290626 to a new imbalance of 0.016537504534195144. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023985592383247666); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03746753177284661, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.291, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023985592383247666); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043666666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:54:01,362 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1295118860:76 , srv90362807:77 , srv1814025136:78 , srv1897683111:80 , srv1904935497:80 , srv220021143:80 , srv454672209:80 , srv1902836750:81 , srv395822664:81 , srv140873209:82 , srv1582972710:82 , srv1601606785:82 , srv1645066627:82 , srv255179497:82 , srv375847328:82 , srv443280134:82 , srv906908507:82 , srv1032499291:83 , srv1183011824:83 , srv1909345318:83 , srv1940019092:83 , srv671355397:83 , srv74199630:83 , srv831788024:83 , srv1102565795:84 , srv170430479:84 , srv1708420065:84 , srv1792280444:84 , srv2067396665:84 , srv426228459:84 , srv47662640:84 , srv903511414:84 , srv1185870863:85 , srv1232701018:85 , srv1286258689:85 , srv1749230566:85 , srv1908752373:85 , srv2106052951:85 , srv2127966744:85 , srv245127297:85 , srv343628724:85 , srv364739795:85 , srv422348656:85 , srv434284031:85 , srv473667958:85 , srv86167544:85 , srv878595174:85 , srv1166009101:86 , srv118078519:86 , srv1285813645:86 , srv1339653493:86 , srv1500252463:86 , srv15822157:86 , srv1773002901:86 , srv1783706883:86 , srv1866059938:86 , srv1959040317:86 , srv23039918:86 , srv238082475:86 , srv263531332:86 , srv299910445:86 , srv367777460:86 , srv650764615:86 , srv1157743095:87 , srv1257983984:87 , srv1417725961:87 , srv1505164895:87 , srv1505391986:87 , srv1544079401:87 , srv1724418413:87 , srv1910062989:87 , srv221535383:87 , srv328943074:87 , srv477001456:87 , srv754737452:87 , srv874381431:87 , srv1125047991:88 , srv130386475:88 , srv1323692036:88 , srv1335568754:88 , srv1436725875:88 , srv1471026708:88 , srv429867095:88 , srv65957914:88 , srv746802922:88 , srv978780598:88 , srv997565287:88 , srv1734317169:89 , srv1752471825:89 , srv1868786521:89 , srv2055205135:89 , srv43000455:89 , srv320357861:90 , srv350352895:90 , srv414083758:90 , srv604763457:90 , srv646400019:90 , srv1111852006:91 , srv157512818:91 , srv1493186142:551 } 2023-02-17 20:54:01,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 20:54:01,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2127966744=55, srv1102565795=1, srv454672209=79, srv422348656=73, srv903511414=95, srv245127297=60, srv15822157=28, srv1959040317=51, srv90362807=96, srv1582972710=29, srv1417725961=19, srv2106052951=54, srv343628724=66, srv1285813645=11, srv1335568754=16, srv1910062989=49, srv263531332=62, srv375847328=70, srv746802922=89, srv874381431=93, srv1902836750=45, srv157512818=27, srv1749230566=36, srv1493186142=22, srv1792280444=40, srv1908752373=47, srv238082475=59, srv367777460=69, srv1814025136=41, srv1295118860=13, srv1866059938=42, srv350352895=67, srv1724418413=34, srv414083758=72, srv434284031=77, srv426228459=74, srv650764615=85, srv1157743095=4, srv1232701018=9, srv395822664=71, srv473667958=80, srv1183011824=7, srv1645066627=31, srv130386475=14, srv1166009101=5, srv1868786521=43, srv1783706883=39, srv1752471825=37, srv118078519=6, srv1323692036=15, srv646400019=84, srv1032499291=0, srv65957914=86, srv671355397=87, srv906908507=97, srv86167544=92, srv1940019092=50, srv299910445=63, srv74199630=88, srv221535383=57, srv429867095=75, srv140873209=18, srv1505391986=25, srv2067396665=53, srv604763457=83, srv1544079401=26, srv1257983984=10, srv1904935497=46, srv1471026708=21, srv2055205135=52, srv220021143=56, srv443280134=78, srv255179497=61, srv878595174=94, srv1773002901=38, srv328943074=65, srv1505164895=24, srv320357861=64, srv754737452=90, srv1500252463=23, srv1125047991=3, srv1601606785=30, srv364739795=68, srv170430479=32, srv47662640=81, srv1286258689=12, srv43000455=76, srv477001456=82, srv1339653493=17, srv23039918=58, srv997565287=99, srv1897683111=44, srv1909345318=48, srv1436725875=20, srv1708420065=33, srv1734317169=35, srv831788024=91, srv978780598=98, srv1185870863=8, srv1111852006=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 20:54:01,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 20:54:01,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 20:54:01,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 20:54:01,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 20:54:01,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 20:54:01,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 20:54:01,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 20:54:01,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 20:54:01,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 20:54:01,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 20:54:01,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 20:54:01,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 20:54:01,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 20:54:01,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 20:54:01,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 20:54:01,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 20:54:01,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 20:54:01,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 20:54:01,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 20:54:01,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 20:54:01,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 20:54:01,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 20:54:01,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 20:54:01,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 20:54:01,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 20:54:01,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 20:54:01,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 20:54:01,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 20:54:01,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 20:54:01,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 20:54:01,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-17 20:54:01,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-17 20:54:01,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-17 20:54:01,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-17 20:54:01,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-17 20:54:01,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-17 20:54:01,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-17 20:54:01,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-17 20:54:01,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-17 20:54:01,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-17 20:54:01,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-17 20:54:01,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-17 20:54:01,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-17 20:54:01,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-17 20:54:01,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-17 20:54:01,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-17 20:54:01,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-17 20:54:01,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-17 20:54:01,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-17 20:54:01,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-17 20:54:01,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-17 20:54:01,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-17 20:54:01,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-17 20:54:01,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-17 20:54:01,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-17 20:54:01,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-17 20:54:01,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-17 20:54:01,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-17 20:54:01,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-17 20:54:01,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-17 20:54:01,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-17 20:54:01,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-17 20:54:01,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-17 20:54:01,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-17 20:54:01,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-17 20:54:01,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-17 20:54:01,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-17 20:54:01,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-17 20:54:01,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-17 20:54:01,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-17 20:54:01,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-17 20:54:01,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-17 20:54:01,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-17 20:54:01,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-17 20:54:01,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-17 20:54:01,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-17 20:54:01,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-17 20:54:01,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-17 20:54:01,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-17 20:54:01,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-17 20:54:01,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-17 20:54:01,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-17 20:54:01,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-17 20:54:01,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-17 20:54:01,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-17 20:54:01,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-17 20:54:01,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-17 20:54:01,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-17 20:54:01,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-17 20:54:01,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-17 20:54:01,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-17 20:54:01,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-17 20:54:01,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-17 20:54:01,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-17 20:54:01,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-17 20:54:01,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-17 20:54:01,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-17 20:54:01,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-17 20:54:01,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-17 20:54:01,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-17 20:54:01,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-17 20:54:01,561 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023985592383247666); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03746753177284661, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023985592383247666); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043666666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:54:01,561 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-17 20:54:01,561 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016519165904235748, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023985592383247666); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03746753177284661, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023985592383247666); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043666666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-17 20:54:01,780 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 364 ms to try 5000 different iterations. Found a solution that moves 997 regions; Going from a computed imbalance of 0.016519165904235748 to a new imbalance of 5.069972675495527E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0069001392512032505); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008697433087526973); MoveCostFunction : (multiplier=7.0, imbalance=0.11077777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0069001392512032505); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:54:01,782 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv157512818:81 , srv1910062989:81 , srv1500252463:82 , srv414083758:82 , srv1111852006:83 , srv1505391986:83 , srv15822157:83 , srv1601606785:83 , srv170430479:83 , srv1734317169:83 , srv2067396665:83 , srv746802922:83 , srv754737452:83 , srv997565287:83 , srv118078519:84 , srv1257983984:84 , srv1544079401:84 , srv1724418413:84 , srv1783706883:84 , srv2055205135:84 , srv255179497:84 , srv350352895:84 , srv429867095:84 , srv43000455:84 , srv604763457:84 , srv646400019:84 , srv74199630:84 , srv978780598:84 , srv140873209:85 , srv1436725875:85 , srv1471026708:85 , srv1752471825:85 , srv1773002901:85 , srv1866059938:85 , srv1959040317:85 , srv2106052951:85 , srv23039918:85 , srv238082475:85 , srv245127297:85 , srv263531332:85 , srv320357861:85 , srv328943074:85 , srv375847328:85 , srv434284031:85 , srv477001456:85 , srv671355397:85 , srv874381431:85 , srv878595174:85 , srv1125047991:86 , srv1183011824:86 , srv1185870863:86 , srv130386475:86 , srv1323692036:86 , srv1335568754:86 , srv1339653493:86 , srv1505164895:86 , srv1868786521:86 , srv1902836750:86 , srv1908752373:86 , srv1909345318:86 , srv1940019092:86 , srv299910445:86 , srv343628724:86 , srv364739795:86 , srv367777460:86 , srv422348656:86 , srv473667958:86 , srv47662640:86 , srv65957914:86 , srv831788024:86 , srv903511414:86 , srv1032499291:87 , srv1102565795:87 , srv1157743095:87 , srv1166009101:87 , srv1417725961:87 , srv1708420065:87 , srv1749230566:87 , srv1897683111:87 , srv2127966744:87 , srv220021143:87 , srv221535383:87 , srv395822664:87 , srv86167544:87 , srv906908507:87 , srv1232701018:88 , srv1285813645:88 , srv1295118860:88 , srv1792280444:88 , srv1904935497:88 , srv426228459:88 , srv443280134:88 , srv650764615:88 , srv1286258689:89 , srv1582972710:89 , srv1645066627:89 , srv90362807:89 , srv1814025136:90 , srv454672209:90 , srv1493186142:532 } 2023-02-17 20:54:01,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 20:54:01,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2127966744=55, srv1102565795=1, srv454672209=79, srv422348656=73, srv903511414=95, srv245127297=60, srv15822157=28, srv1959040317=51, srv90362807=96, srv1582972710=29, srv1417725961=19, srv2106052951=54, srv343628724=66, srv1285813645=11, srv1335568754=16, srv1910062989=49, srv263531332=62, srv375847328=70, srv746802922=89, srv874381431=93, srv1902836750=45, srv157512818=27, srv1749230566=36, srv1493186142=22, srv1792280444=40, srv1908752373=47, srv238082475=59, srv367777460=69, srv1814025136=41, srv1295118860=13, srv1866059938=42, srv350352895=67, srv1724418413=34, srv414083758=72, srv434284031=77, srv426228459=74, srv650764615=85, srv1157743095=4, srv1232701018=9, srv395822664=71, srv473667958=80, srv1183011824=7, srv1645066627=31, srv130386475=14, srv1166009101=5, srv1868786521=43, srv1783706883=39, srv1752471825=37, srv118078519=6, srv1323692036=15, srv646400019=84, srv1032499291=0, srv65957914=86, srv671355397=87, srv906908507=97, srv86167544=92, srv1940019092=50, srv299910445=63, srv74199630=88, srv221535383=57, srv429867095=75, srv140873209=18, srv1505391986=25, srv2067396665=53, srv604763457=83, srv1544079401=26, srv1257983984=10, srv1904935497=46, srv1471026708=21, srv2055205135=52, srv220021143=56, srv443280134=78, srv255179497=61, srv878595174=94, srv1773002901=38, srv328943074=65, srv1505164895=24, srv320357861=64, srv754737452=90, srv1500252463=23, srv1125047991=3, srv1601606785=30, srv364739795=68, srv170430479=32, srv47662640=81, srv1286258689=12, srv43000455=76, srv477001456=82, srv1339653493=17, srv23039918=58, srv997565287=99, srv1897683111=44, srv1909345318=48, srv1436725875=20, srv1708420065=33, srv1734317169=35, srv831788024=91, srv978780598=98, srv1185870863=8, srv1111852006=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 20:54:01,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 20:54:01,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 20:54:01,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 20:54:01,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 20:54:01,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 20:54:01,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 20:54:01,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 20:54:01,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 20:54:01,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 20:54:01,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 20:54:01,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 20:54:01,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 20:54:01,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 20:54:01,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 20:54:01,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 20:54:01,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 20:54:01,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-17 20:54:01,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-17 20:54:01,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-17 20:54:01,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-17 20:54:01,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-17 20:54:01,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-17 20:54:01,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-17 20:54:01,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-17 20:54:01,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-17 20:54:01,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-17 20:54:01,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-17 20:54:01,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-17 20:54:01,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-17 20:54:01,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-17 20:54:01,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-17 20:54:01,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-17 20:54:01,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-17 20:54:01,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-17 20:54:01,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-17 20:54:01,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-17 20:54:01,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-17 20:54:01,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-17 20:54:01,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-17 20:54:01,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-17 20:54:01,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-17 20:54:01,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-17 20:54:01,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-17 20:54:01,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-17 20:54:01,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-17 20:54:01,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-17 20:54:01,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-17 20:54:01,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-17 20:54:01,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-17 20:54:01,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-17 20:54:01,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-17 20:54:01,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-17 20:54:01,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-17 20:54:01,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-17 20:54:01,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-17 20:54:01,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-17 20:54:01,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-17 20:54:01,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-17 20:54:01,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-17 20:54:01,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-17 20:54:01,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-17 20:54:01,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-17 20:54:01,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-17 20:54:01,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-17 20:54:01,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-17 20:54:01,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-17 20:54:01,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-17 20:54:01,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-17 20:54:01,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-17 20:54:01,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-17 20:54:01,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-17 20:54:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-17 20:54:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-17 20:54:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-17 20:54:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-17 20:54:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-17 20:54:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-17 20:54:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-17 20:54:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-17 20:54:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-17 20:54:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-17 20:54:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-17 20:54:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-17 20:54:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-17 20:54:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-17 20:54:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-17 20:54:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-17 20:54:01,922 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0069001392512032505); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008697433087526973); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0069001392512032505); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:54:01,922 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-17 20:54:01,923 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.00016124338587E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0069001392512032505); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008697433087526973); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0069001392512032505); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-17 20:54:02,277 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 458 ms to try 5000 different iterations. Found a solution that moves 261 regions; Going from a computed imbalance of 5.00016124338587E-4 to a new imbalance of 4.3042520363912433E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.116465361813114E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.029, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.116465361813114E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:54:02,278 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1032499291:83 , srv1102565795:83 , srv1257983984:83 , srv130386475:83 , srv1645066627:83 , srv1792280444:83 , srv221535383:83 , srv426228459:83 , srv47662640:83 , srv650764615:83 , srv1111852006:84 , srv1157743095:84 , srv118078519:84 , srv1232701018:84 , srv1285813645:84 , srv1286258689:84 , srv1295118860:84 , srv1323692036:84 , srv140873209:84 , srv1417725961:84 , srv15822157:84 , srv1708420065:84 , srv1749230566:84 , srv1814025136:84 , srv1866059938:84 , srv1868786521:84 , srv1902836750:84 , srv1909345318:84 , srv2067396665:84 , srv2106052951:84 , srv220021143:84 , srv343628724:84 , srv414083758:84 , srv434284031:84 , srv454672209:84 , srv473667958:84 , srv477001456:84 , srv646400019:84 , srv746802922:84 , srv754737452:84 , srv86167544:84 , srv874381431:84 , srv90362807:84 , srv1125047991:85 , srv1185870863:85 , srv1335568754:85 , srv1339653493:85 , srv1436725875:85 , srv1471026708:85 , srv1505391986:85 , srv1544079401:85 , srv157512818:85 , srv1601606785:85 , srv1724418413:85 , srv1734317169:85 , srv1752471825:85 , srv1773002901:85 , srv1783706883:85 , srv1897683111:85 , srv1904935497:85 , srv1908752373:85 , srv1910062989:85 , srv1940019092:85 , srv1959040317:85 , srv2055205135:85 , srv23039918:85 , srv263531332:85 , srv299910445:85 , srv320357861:85 , srv328943074:85 , srv350352895:85 , srv367777460:85 , srv375847328:85 , srv395822664:85 , srv429867095:85 , srv43000455:85 , srv443280134:85 , srv604763457:85 , srv65957914:85 , srv671355397:85 , srv74199630:85 , srv831788024:85 , srv878595174:85 , srv978780598:85 , srv997565287:85 , srv1166009101:86 , srv1183011824:86 , srv1500252463:86 , srv1505164895:86 , srv1582972710:86 , srv170430479:86 , srv2127966744:86 , srv245127297:86 , srv255179497:86 , srv364739795:86 , srv422348656:86 , srv903511414:86 , srv906908507:86 , srv238082475:87 , srv1493186142:623 } 2023-02-17 20:54:02,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 20:54:02,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2127966744=55, srv1102565795=1, srv454672209=79, srv422348656=73, srv903511414=95, srv245127297=60, srv15822157=28, srv1959040317=51, srv90362807=96, srv1582972710=29, srv1417725961=19, srv2106052951=54, srv343628724=66, srv1285813645=11, srv1335568754=16, srv1910062989=49, srv263531332=62, srv375847328=70, srv746802922=89, srv874381431=93, srv1902836750=45, srv157512818=27, srv1749230566=36, srv1493186142=22, srv1792280444=40, srv1908752373=47, srv238082475=59, srv367777460=69, srv1814025136=41, srv1295118860=13, srv1866059938=42, srv350352895=67, srv1724418413=34, srv414083758=72, srv434284031=77, srv426228459=74, srv650764615=85, srv1157743095=4, srv1232701018=9, srv395822664=71, srv473667958=80, srv1183011824=7, srv1645066627=31, srv130386475=14, srv1166009101=5, srv1868786521=43, srv1783706883=39, srv1752471825=37, srv118078519=6, srv1323692036=15, srv646400019=84, srv1032499291=0, srv65957914=86, srv671355397=87, srv906908507=97, srv86167544=92, srv1940019092=50, srv299910445=63, srv74199630=88, srv221535383=57, srv429867095=75, srv140873209=18, srv1505391986=25, srv2067396665=53, srv604763457=83, srv1544079401=26, srv1257983984=10, srv1904935497=46, srv1471026708=21, srv2055205135=52, srv220021143=56, srv443280134=78, srv255179497=61, srv878595174=94, srv1773002901=38, srv328943074=65, srv1505164895=24, srv320357861=64, srv754737452=90, srv1500252463=23, srv1125047991=3, srv1601606785=30, srv364739795=68, srv170430479=32, srv47662640=81, srv1286258689=12, srv43000455=76, srv477001456=82, srv1339653493=17, srv23039918=58, srv997565287=99, srv1897683111=44, srv1909345318=48, srv1436725875=20, srv1708420065=33, srv1734317169=35, srv831788024=91, srv978780598=98, srv1185870863=8, srv1111852006=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 20:54:02,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 20:54:02,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 20:54:02,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 20:54:02,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 20:54:02,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 20:54:02,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 20:54:02,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-17 20:54:02,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-17 20:54:02,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-17 20:54:02,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-17 20:54:02,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-17 20:54:02,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-17 20:54:02,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-17 20:54:02,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-17 20:54:02,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-17 20:54:02,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-17 20:54:02,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-17 20:54:02,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-17 20:54:02,399 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.116465361813114E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.116465361813114E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:54:02,400 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-17 20:54:02,400 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.1214959302666636E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.116465361813114E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.116465361813114E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-17 20:54:02,612 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 305 ms to try 5000 different iterations. Found a solution that moves 30 regions; Going from a computed imbalance of 4.1214959302666636E-5 to a new imbalance of 7.5978219901109264E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0033333333333333335); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:54:02,613 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv170430479:82 , srv1166009101:83 , srv1185870863:83 , srv1232701018:83 , srv1505164895:83 , srv1505391986:83 , srv1582972710:83 , srv238082475:83 , srv245127297:83 , srv320357861:83 , srv422348656:83 , srv874381431:83 , srv906908507:83 , srv1032499291:84 , srv1102565795:84 , srv1111852006:84 , srv1125047991:84 , srv1157743095:84 , srv1183011824:84 , srv1285813645:84 , srv1286258689:84 , srv1323692036:84 , srv1335568754:84 , srv1339653493:84 , srv1417725961:84 , srv1436725875:84 , srv1471026708:84 , srv1500252463:84 , srv1544079401:84 , srv15822157:84 , srv1601606785:84 , srv1645066627:84 , srv1708420065:84 , srv1724418413:84 , srv1734317169:84 , srv1749230566:84 , srv1752471825:84 , srv1773002901:84 , srv1783706883:84 , srv1792280444:84 , srv1814025136:84 , srv1868786521:84 , srv1897683111:84 , srv1902836750:84 , srv1904935497:84 , srv1908752373:84 , srv1909345318:84 , srv1910062989:84 , srv1940019092:84 , srv1959040317:84 , srv2055205135:84 , srv2127966744:84 , srv220021143:84 , srv221535383:84 , srv23039918:84 , srv255179497:84 , srv299910445:84 , srv328943074:84 , srv350352895:84 , srv364739795:84 , srv367777460:84 , srv375847328:84 , srv395822664:84 , srv414083758:84 , srv429867095:84 , srv43000455:84 , srv434284031:84 , srv443280134:84 , srv454672209:84 , srv473667958:84 , srv47662640:84 , srv477001456:84 , srv604763457:84 , srv646400019:84 , srv650764615:84 , srv65957914:84 , srv671355397:84 , srv74199630:84 , srv746802922:84 , srv831788024:84 , srv86167544:84 , srv878595174:84 , srv903511414:84 , srv978780598:84 , srv997565287:84 , srv118078519:85 , srv1257983984:85 , srv1295118860:85 , srv130386475:85 , srv140873209:85 , srv157512818:85 , srv1866059938:85 , srv2067396665:85 , srv2106052951:85 , srv263531332:85 , srv343628724:85 , srv426228459:85 , srv754737452:85 , srv90362807:85 , srv1493186142:684 } 2023-02-17 20:54:02,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 20:54:02,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2127966744=55, srv1102565795=1, srv454672209=79, srv422348656=73, srv903511414=95, srv245127297=60, srv15822157=28, srv1959040317=51, srv90362807=96, srv1582972710=29, srv1417725961=19, srv2106052951=54, srv343628724=66, srv1285813645=11, srv1335568754=16, srv1910062989=49, srv263531332=62, srv375847328=70, srv746802922=89, srv874381431=93, srv1902836750=45, srv157512818=27, srv1749230566=36, srv1493186142=22, srv1792280444=40, srv1908752373=47, srv238082475=59, srv367777460=69, srv1814025136=41, srv1295118860=13, srv1866059938=42, srv350352895=67, srv1724418413=34, srv414083758=72, srv434284031=77, srv426228459=74, srv650764615=85, srv1157743095=4, srv1232701018=9, srv395822664=71, srv473667958=80, srv1183011824=7, srv1645066627=31, srv130386475=14, srv1166009101=5, srv1868786521=43, srv1783706883=39, srv1752471825=37, srv118078519=6, srv1323692036=15, srv646400019=84, srv1032499291=0, srv65957914=86, srv671355397=87, srv906908507=97, srv86167544=92, srv1940019092=50, srv299910445=63, srv74199630=88, srv221535383=57, srv429867095=75, srv140873209=18, srv1505391986=25, srv2067396665=53, srv604763457=83, srv1544079401=26, srv1257983984=10, srv1904935497=46, srv1471026708=21, srv2055205135=52, srv220021143=56, srv443280134=78, srv255179497=61, srv878595174=94, srv1773002901=38, srv328943074=65, srv1505164895=24, srv320357861=64, srv754737452=90, srv1500252463=23, srv1125047991=3, srv1601606785=30, srv364739795=68, srv170430479=32, srv47662640=81, srv1286258689=12, srv43000455=76, srv477001456=82, srv1339653493=17, srv23039918=58, srv997565287=99, srv1897683111=44, srv1909345318=48, srv1436725875=20, srv1708420065=33, srv1734317169=35, srv831788024=91, srv978780598=98, srv1185870863=8, srv1111852006=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 20:54:02,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 20:54:02,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 20:54:02,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 20:54:02,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 20:54:02,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 20:54:02,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 20:54:02,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 20:54:02,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 20:54:02,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 20:54:02,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 20:54:02,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 20:54:02,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 20:54:02,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 20:54:02,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 20:54:02,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 20:54:02,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 20:54:02,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 20:54:02,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 20:54:02,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 20:54:02,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 20:54:02,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 20:54:02,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 20:54:02,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 20:54:02,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 20:54:02,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 20:54:02,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 20:54:02,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 20:54:02,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 20:54:02,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 20:54:02,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 20:54:02,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-17 20:54:02,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-17 20:54:02,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-17 20:54:02,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-17 20:54:02,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-17 20:54:02,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-17 20:54:02,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-17 20:54:02,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-17 20:54:02,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-17 20:54:02,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-17 20:54:02,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-17 20:54:02,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-17 20:54:02,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-17 20:54:02,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-17 20:54:02,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-17 20:54:02,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-17 20:54:02,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-17 20:54:02,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-17 20:54:02,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-17 20:54:02,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-17 20:54:02,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-17 20:54:02,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-17 20:54:02,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-17 20:54:02,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-17 20:54:02,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-17 20:54:02,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-17 20:54:02,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-17 20:54:02,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-17 20:54:02,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-17 20:54:02,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-17 20:54:02,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-17 20:54:02,765 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:54:02,765 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv170430479:82 , srv1166009101:83 , srv1185870863:83 , srv1232701018:83 , srv1505164895:83 , srv1505391986:83 , srv1582972710:83 , srv238082475:83 , srv245127297:83 , srv320357861:83 , srv422348656:83 , srv874381431:83 , srv906908507:83 , srv1032499291:84 , srv1102565795:84 , srv1111852006:84 , srv1125047991:84 , srv1157743095:84 , srv1183011824:84 , srv1285813645:84 , srv1286258689:84 , srv1323692036:84 , srv1335568754:84 , srv1339653493:84 , srv1417725961:84 , srv1436725875:84 , srv1471026708:84 , srv1500252463:84 , srv1544079401:84 , srv15822157:84 , srv1601606785:84 , srv1645066627:84 , srv1708420065:84 , srv1724418413:84 , srv1734317169:84 , srv1749230566:84 , srv1752471825:84 , srv1773002901:84 , srv1783706883:84 , srv1792280444:84 , srv1814025136:84 , srv1868786521:84 , srv1897683111:84 , srv1902836750:84 , srv1904935497:84 , srv1908752373:84 , srv1909345318:84 , srv1910062989:84 , srv1940019092:84 , srv1959040317:84 , srv2055205135:84 , srv2127966744:84 , srv220021143:84 , srv221535383:84 , srv23039918:84 , srv255179497:84 , srv299910445:84 , srv328943074:84 , srv350352895:84 , srv364739795:84 , srv367777460:84 , srv375847328:84 , srv395822664:84 , srv414083758:84 , srv429867095:84 , srv43000455:84 , srv434284031:84 , srv443280134:84 , srv454672209:84 , srv473667958:84 , srv47662640:84 , srv477001456:84 , srv604763457:84 , srv646400019:84 , srv650764615:84 , srv65957914:84 , srv671355397:84 , srv74199630:84 , srv746802922:84 , srv831788024:84 , srv86167544:84 , srv878595174:84 , srv903511414:84 , srv978780598:84 , srv997565287:84 , srv118078519:85 , srv1257983984:85 , srv1295118860:85 , srv130386475:85 , srv140873209:85 , srv157512818:85 , srv1866059938:85 , srv2067396665:85 , srv2106052951:85 , srv263531332:85 , srv343628724:85 , srv426228459:85 , srv754737452:85 , srv90362807:85 , srv1493186142:684 } 2023-02-17 20:54:02,789 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=372 (was 369) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=7831 (was 8336) 2023-02-17 20:54:02,796 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=372, ProcessCount=171, AvailableMemoryMB=7831 2023-02-17 20:54:02,797 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-17 20:54:02,797 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-02-17 20:54:02,808 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1081209334:84 , srv1146542159:84 , srv1194356253:84 , srv1204418370:84 , srv1216688553:84 , srv122479432:84 , srv1230748228:84 , srv1249882591:84 , srv1451791966:84 , srv1513869170:84 , srv154617750:84 , srv1600646922:84 , srv1607514780:84 , srv1628466933:84 , srv1651130376:84 , srv1856473877:84 , srv2012253986:84 , srv2049174934:84 , srv252219547:84 , srv426493991:84 , srv511184398:84 , srv609165753:84 , srv698395859:84 , srv756259017:84 , srv779010966:84 , srv780386674:84 , srv820173356:84 , srv851177260:84 , srv991080936:84 , srv443155309:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-02-17 20:54:02,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 20:54:02,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1628466933=13, srv1600646922=11, srv154617750=10, srv252219547=18, srv779010966=25, srv443155309=20, srv2012253986=16, srv2049174934=17, srv426493991=19, srv780386674=26, srv1194356253=2, srv609165753=22, srv1216688553=4, srv1513869170=9, srv698395859=23, srv1451791966=8, srv511184398=21, srv851177260=28, srv820173356=27, srv1081209334=0, srv1146542159=1, srv991080936=29, srv1651130376=14, srv1607514780=12, srv1249882591=7, srv1230748228=6, srv1856473877=15, srv756259017=24, srv1204418370=3, srv122479432=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 20:54:02,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 20:54:02,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 20:54:02,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 20:54:02,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 20:54:02,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 20:54:02,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 20:54:02,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 20:54:02,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-17 20:54:02,839 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1762789528436275, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:54:02,839 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912748492488651, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1762789528436275, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-02-17 20:56:02,825 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6396372 different iterations. Found a solution that moves 1848 regions; Going from a computed imbalance of 0.9912748492488651 to a new imbalance of 4.313324190526491E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:56:02,833 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1081209334:90 , srv1146542159:90 , srv1194356253:90 , srv1204418370:90 , srv1216688553:90 , srv122479432:90 , srv1230748228:90 , srv1249882591:90 , srv1451791966:90 , srv1513869170:90 , srv154617750:90 , srv1600646922:90 , srv1607514780:90 , srv1628466933:90 , srv1651130376:90 , srv1856473877:90 , srv2012253986:90 , srv2049174934:90 , srv252219547:90 , srv426493991:90 , srv443155309:90 , srv511184398:90 , srv609165753:90 , srv698395859:90 , srv756259017:90 , srv779010966:90 , srv780386674:90 , srv820173356:90 , srv851177260:90 , srv991080936:90 } 2023-02-17 20:56:02,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 20:56:02,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1628466933=13, srv1600646922=11, srv154617750=10, srv252219547=18, srv779010966=25, srv443155309=20, srv2012253986=16, srv2049174934=17, srv426493991=19, srv780386674=26, srv1194356253=2, srv609165753=22, srv1216688553=4, srv1513869170=9, srv698395859=23, srv1451791966=8, srv511184398=21, srv851177260=28, srv820173356=27, srv1081209334=0, srv1146542159=1, srv991080936=29, srv1651130376=14, srv1607514780=12, srv1249882591=7, srv1230748228=6, srv1856473877=15, srv756259017=24, srv1204418370=3, srv122479432=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 20:56:02,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 20:56:02,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 20:56:02,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-17 20:56:02,856 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 20:56:02,857 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1081209334:90 , srv1146542159:90 , srv1194356253:90 , srv1204418370:90 , srv1216688553:90 , srv122479432:90 , srv1230748228:90 , srv1249882591:90 , srv1451791966:90 , srv1513869170:90 , srv154617750:90 , srv1600646922:90 , srv1607514780:90 , srv1628466933:90 , srv1651130376:90 , srv1856473877:90 , srv2012253986:90 , srv2049174934:90 , srv252219547:90 , srv426493991:90 , srv443155309:90 , srv511184398:90 , srv609165753:90 , srv698395859:90 , srv756259017:90 , srv779010966:90 , srv780386674:90 , srv820173356:90 , srv851177260:90 , srv991080936:90 } 2023-02-17 20:56:02,877 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=290 (was 372), ProcessCount=173 (was 171) - ProcessCount LEAK? -, AvailableMemoryMB=7643 (was 7831)