2023-08-23 01:53:12,474 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-23 01:53:12,791 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-23 01:53:12,905 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-23 01:53:12,977 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-23 01:53:12,984 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-23 01:53:12,998 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=213, ProcessCount=171, AvailableMemoryMB=12366 2023-08-23 01:53:13,000 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-23 01:53:13,000 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-23 01:53:13,140 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1019809375:84 , srv1035116893:84 , srv1084260722:84 , srv1092857921:84 , srv1120854884:84 , srv1127495963:84 , srv1144854551:84 , srv1175824347:84 , srv1203898101:84 , srv1257678659:84 , srv126266550:84 , srv127600704:84 , srv1329136819:84 , srv1370854964:84 , srv1411265639:84 , srv1440150290:84 , srv1442193722:84 , srv1471702152:84 , srv1490233556:84 , srv1492924035:84 , srv1519301605:84 , srv1543938853:84 , srv1559461113:84 , srv1567442895:84 , srv1580640106:84 , srv1584016614:84 , srv1600922175:84 , srv1616581851:84 , srv168956454:84 , srv1738665106:84 , srv1787327500:84 , srv1793395269:84 , srv1806363812:84 , srv1854233040:84 , srv1854254300:84 , srv1857421453:84 , srv1864031301:84 , srv1866989004:84 , srv1878899026:84 , srv1882816802:84 , srv1885261242:84 , srv1901880297:84 , srv1924113130:84 , srv1936385292:84 , srv1944261833:84 , srv1955624000:84 , srv1967393235:84 , srv1994672933:84 , srv1995618427:84 , srv2005375347:84 , srv2007370742:84 , srv2027798578:84 , srv2030975885:84 , srv2084235363:84 , srv2129999758:84 , srv216890186:84 , srv229845845:84 , srv244944334:84 , srv273700671:84 , srv284504755:84 , srv290808437:84 , srv306877577:84 , srv308292653:84 , srv319508232:84 , srv347715075:84 , srv356214082:84 , srv358555660:84 , srv370776968:84 , srv391784551:84 , srv406858178:84 , srv411577880:84 , srv420340824:84 , srv428140188:84 , srv446971066:84 , srv45765517:84 , srv473551831:84 , srv482931484:84 , srv533439874:84 , srv576058804:84 , srv592877183:84 , srv603659469:84 , srv664868010:84 , srv676449597:84 , srv696957234:84 , srv742721408:84 , srv745436037:84 , srv757316437:84 , srv779572094:84 , srv837494862:84 , srv850240421:84 , srv862361609:84 , srv867185234:84 , srv875179863:84 , srv91011834:84 , srv921056893:84 , srv958266480:84 , srv960729498:84 , srv965325346:84 , srv967917557:84 , srv1076062989:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-23 01:53:13,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 01:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv273700671=59, srv2007370742=51, srv347715075=65, srv428140188=73, srv837494862=89, srv1175824347=8, srv1600922175=27, srv603659469=81, srv1854233040=34, srv1793395269=32, srv229845845=57, srv1019809375=0, srv1854254300=35, srv1967393235=47, srv2005375347=50, srv960729498=97, srv391784551=69, srv1543938853=22, srv290808437=61, srv1092857921=4, srv1329136819=13, srv592877183=80, srv745436037=86, srv1806363812=33, srv473551831=76, srv446971066=74, srv1440150290=16, srv2129999758=55, srv127600704=12, srv1120854884=5, srv757316437=87, srv411577880=71, srv356214082=66, srv2027798578=52, srv1411265639=15, srv1878899026=39, srv1882816802=40, srv1471702152=18, srv406858178=70, srv1944261833=45, srv370776968=68, srv742721408=85, srv1035116893=1, srv420340824=72, srv875179863=93, srv284504755=60, srv850240421=90, srv1885261242=41, srv1559461113=23, srv308292653=63, srv216890186=56, srv45765517=75, srv1936385292=44, srv319508232=64, srv1995618427=49, srv779572094=88, srv1492924035=20, srv664868010=82, srv1924113130=43, srv1490233556=19, srv358555660=67, srv862361609=91, srv1076062989=2, srv1144854551=7, srv1084260722=3, srv533439874=78, srv1127495963=6, srv126266550=11, srv921056893=95, srv1864031301=37, srv1994672933=48, srv91011834=94, srv867185234=92, srv1442193722=17, srv1616581851=28, srv244944334=58, srv1370854964=14, srv1955624000=46, srv2084235363=54, srv306877577=62, srv576058804=79, srv482931484=77, srv958266480=96, srv965325346=98, srv1203898101=9, srv1787327500=31, srv168956454=29, srv2030975885=53, srv676449597=83, srv1584016614=26, srv1857421453=36, srv1738665106=30, srv1519301605=21, srv1901880297=42, srv1866989004=38, srv1257678659=10, srv1580640106=25, srv967917557=99, srv1567442895=24, srv696957234=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 01:53:13,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 01:53:13,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 01:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 01:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 01:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 01:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 01:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 01:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 01:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 01:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 01:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 01:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 01:53:13,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 01:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 01:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 01:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 01:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 01:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 01:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 01:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 01:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 01:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 01:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 01:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 01:53:13,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 01:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 01:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 01:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 01:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 01:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 01:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-23 01:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-23 01:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-23 01:53:13,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-23 01:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-23 01:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-23 01:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-23 01:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-23 01:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-23 01:53:13,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-23 01:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-23 01:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-23 01:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-23 01:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-23 01:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-23 01:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-23 01:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-23 01:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-23 01:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-23 01:53:13,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-23 01:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-23 01:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-23 01:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-23 01:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-23 01:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-23 01:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-23 01:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-23 01:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-23 01:53:13,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-23 01:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-23 01:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-23 01:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-23 01:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-23 01:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-23 01:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-23 01:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-23 01:53:13,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-23 01:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-23 01:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-23 01:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-23 01:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-23 01:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-23 01:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-23 01:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-23 01:53:13,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-23 01:53:13,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-23 01:53:13,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-23 01:53:13,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-23 01:53:13,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-23 01:53:13,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-23 01:53:13,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-23 01:53:13,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-23 01:53:13,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-23 01:53:13,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-23 01:53:13,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-23 01:53:13,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-23 01:53:13,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-23 01:53:13,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-23 01:53:13,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-23 01:53:13,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-23 01:53:13,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-23 01:53:13,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-23 01:53:13,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-23 01:53:13,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-23 01:53:13,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-23 01:53:13,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-23 01:53:13,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-23 01:53:13,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-23 01:53:13,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-23 01:53:13,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-23 01:53:13,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-23 01:53:13,452 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:53:13,452 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-23 01:53:13,452 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-23 01:53:13,835 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 568 ms to try 5000 different iterations. Found a solution that moves 4726 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17432498012189238. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04228449539655775, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056062335035673254, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5251111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04228449539655775, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16941666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:53:13,847 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1440150290:82 , srv1882816802:82 , srv664868010:82 , srv1035116893:83 , srv1857421453:83 , srv391784551:83 , srv473551831:83 , srv482931484:83 , srv576058804:83 , srv745436037:83 , srv1084260722:84 , srv127600704:84 , srv1559461113:84 , srv1600922175:84 , srv1793395269:84 , srv1806363812:84 , srv1944261833:84 , srv1995618427:84 , srv216890186:84 , srv273700671:84 , srv347715075:84 , srv406858178:84 , srv533439874:84 , srv742721408:84 , srv867185234:84 , srv1203898101:85 , srv1411265639:85 , srv1442193722:85 , srv1490233556:85 , srv1492924035:85 , srv1543938853:85 , srv1787327500:85 , srv1854254300:85 , srv1924113130:85 , srv1967393235:85 , srv2084235363:85 , srv2129999758:85 , srv308292653:85 , srv358555660:85 , srv420340824:85 , srv428140188:85 , srv696957234:85 , srv779572094:85 , srv921056893:85 , srv1144854551:86 , srv1471702152:86 , srv1616581851:86 , srv1854233040:86 , srv1866989004:86 , srv2005375347:86 , srv229845845:86 , srv284504755:86 , srv306877577:86 , srv411577880:86 , srv603659469:86 , srv757316437:86 , srv875179863:86 , srv91011834:86 , srv965325346:86 , srv1175824347:87 , srv1257678659:87 , srv1329136819:87 , srv1519301605:87 , srv1885261242:87 , srv1994672933:87 , srv290808437:87 , srv319508232:87 , srv356214082:87 , srv370776968:87 , srv592877183:87 , srv676449597:87 , srv837494862:87 , srv862361609:87 , srv958266480:87 , srv1120854884:88 , srv1567442895:88 , srv1864031301:88 , srv1955624000:88 , srv2027798578:88 , srv45765517:88 , srv960729498:88 , srv967917557:88 , srv1092857921:89 , srv1127495963:89 , srv126266550:89 , srv1370854964:89 , srv1580640106:89 , srv1584016614:89 , srv1738665106:89 , srv1878899026:89 , srv2007370742:89 , srv1019809375:90 , srv1936385292:90 , srv168956454:91 , srv2030975885:91 , srv446971066:91 , srv1901880297:92 , srv244944334:92 , srv850240421:95 , srv1076062989:466 } 2023-08-23 01:53:13,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 01:53:13,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv273700671=59, srv2007370742=51, srv347715075=65, srv428140188=73, srv837494862=89, srv1175824347=8, srv1600922175=27, srv603659469=81, srv1854233040=34, srv1793395269=32, srv229845845=57, srv1019809375=0, srv1854254300=35, srv1967393235=47, srv2005375347=50, srv960729498=97, srv391784551=69, srv1543938853=22, srv290808437=61, srv1092857921=4, srv1329136819=13, srv592877183=80, srv745436037=86, srv1806363812=33, srv473551831=76, srv446971066=74, srv1440150290=16, srv2129999758=55, srv127600704=12, srv1120854884=5, srv757316437=87, srv411577880=71, srv356214082=66, srv2027798578=52, srv1411265639=15, srv1878899026=39, srv1882816802=40, srv1471702152=18, srv406858178=70, srv1944261833=45, srv370776968=68, srv742721408=85, srv1035116893=1, srv420340824=72, srv875179863=93, srv284504755=60, srv850240421=90, srv1885261242=41, srv1559461113=23, srv308292653=63, srv216890186=56, srv45765517=75, srv1936385292=44, srv319508232=64, srv1995618427=49, srv779572094=88, srv1492924035=20, srv664868010=82, srv1924113130=43, srv1490233556=19, srv358555660=67, srv862361609=91, srv1076062989=2, srv1144854551=7, srv1084260722=3, srv533439874=78, srv1127495963=6, srv126266550=11, srv921056893=95, srv1864031301=37, srv1994672933=48, srv91011834=94, srv867185234=92, srv1442193722=17, srv1616581851=28, srv244944334=58, srv1370854964=14, srv1955624000=46, srv2084235363=54, srv306877577=62, srv576058804=79, srv482931484=77, srv958266480=96, srv965325346=98, srv1203898101=9, srv1787327500=31, srv168956454=29, srv2030975885=53, srv676449597=83, srv1584016614=26, srv1857421453=36, srv1738665106=30, srv1519301605=21, srv1901880297=42, srv1866989004=38, srv1257678659=10, srv1580640106=25, srv967917557=99, srv1567442895=24, srv696957234=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 01:53:13,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 01:53:13,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 01:53:13,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 01:53:13,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 01:53:13,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 01:53:13,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 01:53:13,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 01:53:13,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 01:53:13,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 01:53:13,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 01:53:13,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 01:53:13,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 01:53:13,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 01:53:13,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 01:53:13,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 01:53:13,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 01:53:13,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 01:53:13,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 01:53:13,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 01:53:13,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 01:53:13,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 01:53:13,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 01:53:13,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 01:53:13,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 01:53:13,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 01:53:13,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 01:53:13,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 01:53:13,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 01:53:13,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 01:53:13,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 01:53:13,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-23 01:53:13,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-23 01:53:13,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-23 01:53:13,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-23 01:53:13,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-23 01:53:13,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-23 01:53:13,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-23 01:53:13,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-23 01:53:13,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-23 01:53:13,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-23 01:53:13,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-23 01:53:13,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-23 01:53:13,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-23 01:53:13,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-23 01:53:13,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-23 01:53:13,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-23 01:53:13,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-23 01:53:13,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-23 01:53:13,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-23 01:53:13,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-23 01:53:13,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-23 01:53:13,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-23 01:53:13,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-23 01:53:13,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-23 01:53:13,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-23 01:53:13,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-23 01:53:13,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-23 01:53:13,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-23 01:53:13,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-23 01:53:13,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-23 01:53:13,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-23 01:53:13,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-23 01:53:13,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-23 01:53:13,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-23 01:53:13,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-23 01:53:13,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-23 01:53:13,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-23 01:53:13,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-23 01:53:13,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-23 01:53:13,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-23 01:53:13,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-23 01:53:13,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-23 01:53:13,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-23 01:53:13,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-23 01:53:13,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-23 01:53:13,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-23 01:53:13,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-23 01:53:13,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-23 01:53:13,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-23 01:53:13,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-23 01:53:13,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-23 01:53:13,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-23 01:53:13,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-23 01:53:13,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-23 01:53:13,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-23 01:53:13,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-23 01:53:13,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-23 01:53:13,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-23 01:53:13,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-23 01:53:13,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-23 01:53:13,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-23 01:53:13,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-23 01:53:13,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-23 01:53:13,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-23 01:53:13,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-23 01:53:13,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-23 01:53:13,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-23 01:53:13,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-23 01:53:13,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-23 01:53:13,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-23 01:53:13,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-23 01:53:14,010 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04228449539655775, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056062335035673254, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04228449539655775, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16941666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:53:14,010 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-23 01:53:14,010 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17429188796259945, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04228449539655775, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056062335035673254, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04228449539655775, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16941666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-23 01:53:14,308 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 412 ms to try 5000 different iterations. Found a solution that moves 2541 regions; Going from a computed imbalance of 0.17429188796259945 to a new imbalance of 0.01571342548921132. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023960103181266033); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03454368239296432, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2823333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023960103181266033); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:53:14,320 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv850240421:77 , srv1901880297:79 , srv446971066:79 , srv168956454:80 , srv2030975885:80 , srv1019809375:81 , srv2007370742:81 , srv126266550:82 , srv1370854964:82 , srv1092857921:83 , srv1120854884:83 , srv1127495963:83 , srv1584016614:83 , srv1738665106:83 , srv1864031301:83 , srv1885261242:83 , srv1936385292:83 , srv356214082:83 , srv676449597:83 , srv875179863:83 , srv1519301605:84 , srv1787327500:84 , srv1878899026:84 , srv244944334:84 , srv284504755:84 , srv411577880:84 , srv592877183:84 , srv862361609:84 , srv960729498:84 , srv1144854551:85 , srv1175824347:85 , srv1203898101:85 , srv1543938853:85 , srv1854233040:85 , srv1854254300:85 , srv1866989004:85 , srv1924113130:85 , srv1955624000:85 , srv1994672933:85 , srv2129999758:85 , srv290808437:85 , srv306877577:85 , srv358555660:85 , srv45765517:85 , srv757316437:85 , srv1257678659:86 , srv1329136819:86 , srv1442193722:86 , srv1490233556:86 , srv1492924035:86 , srv1567442895:86 , srv2027798578:86 , srv229845845:86 , srv370776968:86 , srv420340824:86 , srv696957234:86 , srv837494862:86 , srv91011834:86 , srv921056893:86 , srv958266480:86 , srv965325346:86 , srv127600704:87 , srv1411265639:87 , srv1559461113:87 , srv1580640106:87 , srv1616581851:87 , srv1793395269:87 , srv308292653:87 , srv576058804:87 , srv603659469:87 , srv742721408:87 , srv1471702152:88 , srv1600922175:88 , srv1806363812:88 , srv1944261833:88 , srv1967393235:88 , srv1995618427:88 , srv2005375347:88 , srv2084235363:88 , srv216890186:88 , srv273700671:88 , srv319508232:88 , srv347715075:88 , srv428140188:88 , srv473551831:88 , srv779572094:88 , srv967917557:88 , srv1084260722:89 , srv1440150290:89 , srv1882816802:89 , srv391784551:89 , srv482931484:89 , srv664868010:89 , srv867185234:89 , srv1857421453:90 , srv406858178:90 , srv533439874:90 , srv1035116893:91 , srv745436037:91 , srv1076062989:521 } 2023-08-23 01:53:14,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 01:53:14,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv273700671=59, srv2007370742=51, srv347715075=65, srv428140188=73, srv837494862=89, srv1175824347=8, srv1600922175=27, srv603659469=81, srv1854233040=34, srv1793395269=32, srv229845845=57, srv1019809375=0, srv1854254300=35, srv1967393235=47, srv2005375347=50, srv960729498=97, srv391784551=69, srv1543938853=22, srv290808437=61, srv1092857921=4, srv1329136819=13, srv592877183=80, srv745436037=86, srv1806363812=33, srv473551831=76, srv446971066=74, srv1440150290=16, srv2129999758=55, srv127600704=12, srv1120854884=5, srv757316437=87, srv411577880=71, srv356214082=66, srv2027798578=52, srv1411265639=15, srv1878899026=39, srv1882816802=40, srv1471702152=18, srv406858178=70, srv1944261833=45, srv370776968=68, srv742721408=85, srv1035116893=1, srv420340824=72, srv875179863=93, srv284504755=60, srv850240421=90, srv1885261242=41, srv1559461113=23, srv308292653=63, srv216890186=56, srv45765517=75, srv1936385292=44, srv319508232=64, srv1995618427=49, srv779572094=88, srv1492924035=20, srv664868010=82, srv1924113130=43, srv1490233556=19, srv358555660=67, srv862361609=91, srv1076062989=2, srv1144854551=7, srv1084260722=3, srv533439874=78, srv1127495963=6, srv126266550=11, srv921056893=95, srv1864031301=37, srv1994672933=48, srv91011834=94, srv867185234=92, srv1442193722=17, srv1616581851=28, srv244944334=58, srv1370854964=14, srv1955624000=46, srv2084235363=54, srv306877577=62, srv576058804=79, srv482931484=77, srv958266480=96, srv965325346=98, srv1203898101=9, srv1787327500=31, srv168956454=29, srv2030975885=53, srv676449597=83, srv1584016614=26, srv1857421453=36, srv1738665106=30, srv1519301605=21, srv1901880297=42, srv1866989004=38, srv1257678659=10, srv1580640106=25, srv967917557=99, srv1567442895=24, srv696957234=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 01:53:14,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 01:53:14,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 01:53:14,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 01:53:14,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 01:53:14,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 01:53:14,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 01:53:14,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 01:53:14,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 01:53:14,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 01:53:14,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 01:53:14,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 01:53:14,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 01:53:14,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 01:53:14,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 01:53:14,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 01:53:14,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 01:53:14,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 01:53:14,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 01:53:14,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 01:53:14,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 01:53:14,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 01:53:14,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 01:53:14,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 01:53:14,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 01:53:14,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 01:53:14,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 01:53:14,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 01:53:14,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 01:53:14,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 01:53:14,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 01:53:14,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-23 01:53:14,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-23 01:53:14,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-23 01:53:14,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-23 01:53:14,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-23 01:53:14,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-23 01:53:14,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-23 01:53:14,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-23 01:53:14,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-23 01:53:14,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-23 01:53:14,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-23 01:53:14,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-23 01:53:14,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-23 01:53:14,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-23 01:53:14,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-23 01:53:14,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-23 01:53:14,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-23 01:53:14,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-23 01:53:14,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-23 01:53:14,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-23 01:53:14,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-23 01:53:14,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-23 01:53:14,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-23 01:53:14,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-23 01:53:14,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-23 01:53:14,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-23 01:53:14,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-23 01:53:14,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-23 01:53:14,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-23 01:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-23 01:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-23 01:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-23 01:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-23 01:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-23 01:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-23 01:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-23 01:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-23 01:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-23 01:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-23 01:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-23 01:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-23 01:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-23 01:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-23 01:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-23 01:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-23 01:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-23 01:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-23 01:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-23 01:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-23 01:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-23 01:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-23 01:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-23 01:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-23 01:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-23 01:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-23 01:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-23 01:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-23 01:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-23 01:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-23 01:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-23 01:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-23 01:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-23 01:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-23 01:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-23 01:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-23 01:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-23 01:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-23 01:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-23 01:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-23 01:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-23 01:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-23 01:53:14,537 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023960103181266033); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03454368239296432, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023960103181266033); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:53:14,537 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-23 01:53:14,538 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0156956330269254, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023960103181266033); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03454368239296432, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023960103181266033); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-23 01:53:14,802 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 405 ms to try 5000 different iterations. Found a solution that moves 995 regions; Going from a computed imbalance of 0.0156956330269254 to a new imbalance of 8.729519177700488E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008577602156806058); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011537281588060496); MoveCostFunction : (multiplier=7.0, imbalance=0.11055555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008577602156806058); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:53:14,805 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1580640106:80 , srv244944334:81 , srv2005375347:82 , srv319508232:82 , srv967917557:82 , srv1035116893:83 , srv1567442895:83 , srv406858178:83 , srv533439874:83 , srv745436037:83 , srv1084260722:84 , srv1257678659:84 , srv1471702152:84 , srv1857421453:84 , srv1878899026:84 , srv1936385292:84 , srv1967393235:84 , srv2027798578:84 , srv2084235363:84 , srv229845845:84 , srv308292653:84 , srv347715075:84 , srv428140188:84 , srv603659469:84 , srv91011834:84 , srv1127495963:85 , srv1329136819:85 , srv1411265639:85 , srv1442193722:85 , srv1600922175:85 , srv1616581851:85 , srv1738665106:85 , srv1806363812:85 , srv1955624000:85 , srv2030975885:85 , srv290808437:85 , srv370776968:85 , srv45765517:85 , srv837494862:85 , srv862361609:85 , srv867185234:85 , srv960729498:85 , srv965325346:85 , srv1092857921:86 , srv1120854884:86 , srv1144854551:86 , srv1175824347:86 , srv127600704:86 , srv1440150290:86 , srv1490233556:86 , srv1492924035:86 , srv1543938853:86 , srv1559461113:86 , srv1584016614:86 , srv168956454:86 , srv1793395269:86 , srv1854233040:86 , srv1854254300:86 , srv1866989004:86 , srv1882816802:86 , srv1944261833:86 , srv1995618427:86 , srv2129999758:86 , srv216890186:86 , srv306877577:86 , srv356214082:86 , srv358555660:86 , srv391784551:86 , srv411577880:86 , srv420340824:86 , srv473551831:86 , srv482931484:86 , srv576058804:86 , srv592877183:86 , srv664868010:86 , srv742721408:86 , srv757316437:86 , srv850240421:86 , srv921056893:86 , srv958266480:86 , srv1019809375:87 , srv126266550:87 , srv1370854964:87 , srv1864031301:87 , srv1885261242:87 , srv1901880297:87 , srv1924113130:87 , srv1994672933:87 , srv2007370742:87 , srv273700671:87 , srv284504755:87 , srv446971066:87 , srv676449597:87 , srv696957234:87 , srv779572094:87 , srv1203898101:88 , srv1519301605:88 , srv875179863:88 , srv1787327500:90 , srv1076062989:547 } 2023-08-23 01:53:14,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 01:53:14,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv273700671=59, srv2007370742=51, srv347715075=65, srv428140188=73, srv837494862=89, srv1175824347=8, srv1600922175=27, srv603659469=81, srv1854233040=34, srv1793395269=32, srv229845845=57, srv1019809375=0, srv1854254300=35, srv1967393235=47, srv2005375347=50, srv960729498=97, srv391784551=69, srv1543938853=22, srv290808437=61, srv1092857921=4, srv1329136819=13, srv592877183=80, srv745436037=86, srv1806363812=33, srv473551831=76, srv446971066=74, srv1440150290=16, srv2129999758=55, srv127600704=12, srv1120854884=5, srv757316437=87, srv411577880=71, srv356214082=66, srv2027798578=52, srv1411265639=15, srv1878899026=39, srv1882816802=40, srv1471702152=18, srv406858178=70, srv1944261833=45, srv370776968=68, srv742721408=85, srv1035116893=1, srv420340824=72, srv875179863=93, srv284504755=60, srv850240421=90, srv1885261242=41, srv1559461113=23, srv308292653=63, srv216890186=56, srv45765517=75, srv1936385292=44, srv319508232=64, srv1995618427=49, srv779572094=88, srv1492924035=20, srv664868010=82, srv1924113130=43, srv1490233556=19, srv358555660=67, srv862361609=91, srv1076062989=2, srv1144854551=7, srv1084260722=3, srv533439874=78, srv1127495963=6, srv126266550=11, srv921056893=95, srv1864031301=37, srv1994672933=48, srv91011834=94, srv867185234=92, srv1442193722=17, srv1616581851=28, srv244944334=58, srv1370854964=14, srv1955624000=46, srv2084235363=54, srv306877577=62, srv576058804=79, srv482931484=77, srv958266480=96, srv965325346=98, srv1203898101=9, srv1787327500=31, srv168956454=29, srv2030975885=53, srv676449597=83, srv1584016614=26, srv1857421453=36, srv1738665106=30, srv1519301605=21, srv1901880297=42, srv1866989004=38, srv1257678659=10, srv1580640106=25, srv967917557=99, srv1567442895=24, srv696957234=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 01:53:14,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 01:53:14,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 01:53:14,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 01:53:14,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 01:53:14,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 01:53:14,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 01:53:14,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 01:53:14,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 01:53:14,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 01:53:14,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 01:53:14,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 01:53:14,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 01:53:14,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 01:53:14,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 01:53:14,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 01:53:14,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 01:53:14,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 01:53:14,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 01:53:14,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 01:53:14,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 01:53:14,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 01:53:14,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 01:53:14,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 01:53:14,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 01:53:14,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 01:53:14,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 01:53:14,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 01:53:14,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 01:53:14,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 01:53:14,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 01:53:14,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-23 01:53:14,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-23 01:53:14,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-23 01:53:14,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-23 01:53:14,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-23 01:53:14,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-23 01:53:14,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-23 01:53:14,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-23 01:53:14,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-23 01:53:14,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-23 01:53:14,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-23 01:53:14,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-23 01:53:14,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-23 01:53:14,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-23 01:53:14,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-23 01:53:14,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-23 01:53:14,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-23 01:53:14,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-23 01:53:14,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-23 01:53:14,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-23 01:53:14,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-23 01:53:14,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-23 01:53:14,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-23 01:53:14,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-23 01:53:14,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-23 01:53:14,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-23 01:53:14,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-23 01:53:14,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-23 01:53:14,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-23 01:53:14,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-23 01:53:14,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-23 01:53:14,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-23 01:53:14,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-23 01:53:14,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-23 01:53:14,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-23 01:53:14,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-23 01:53:14,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-23 01:53:14,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-23 01:53:14,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-23 01:53:14,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-23 01:53:14,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-23 01:53:14,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-23 01:53:14,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-23 01:53:14,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-23 01:53:14,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-23 01:53:14,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-23 01:53:14,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-23 01:53:14,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-23 01:53:14,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-23 01:53:14,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-23 01:53:14,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-23 01:53:14,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-23 01:53:14,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-23 01:53:14,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-23 01:53:14,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-23 01:53:14,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-23 01:53:14,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-23 01:53:14,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-23 01:53:14,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-23 01:53:14,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-23 01:53:14,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-23 01:53:14,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-23 01:53:14,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-23 01:53:14,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-23 01:53:14,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-23 01:53:14,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-23 01:53:14,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-23 01:53:14,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-23 01:53:14,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-23 01:53:14,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-23 01:53:14,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-23 01:53:14,952 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008577602156806058); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011537281588060496); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008577602156806058); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:53:14,952 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-23 01:53:14,952 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.659847788584031E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008577602156806058); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011537281588060496); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008577602156806058); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-23 01:53:15,407 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 558 ms to try 5000 different iterations. Found a solution that moves 334 regions; Going from a computed imbalance of 8.659847788584031E-4 to a new imbalance of 5.178125852617111E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.03711111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:53:15,408 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv779572094:82 , srv1787327500:83 , srv1994672933:83 , srv273700671:83 , srv958266480:83 , srv1019809375:84 , srv1092857921:84 , srv1144854551:84 , srv1175824347:84 , srv1203898101:84 , srv126266550:84 , srv1329136819:84 , srv1370854964:84 , srv1471702152:84 , srv1519301605:84 , srv1567442895:84 , srv1584016614:84 , srv1616581851:84 , srv1854233040:84 , srv1864031301:84 , srv1901880297:84 , srv1944261833:84 , srv1955624000:84 , srv2027798578:84 , srv306877577:84 , srv370776968:84 , srv391784551:84 , srv446971066:84 , srv45765517:84 , srv482931484:84 , srv533439874:84 , srv592877183:84 , srv664868010:84 , srv696957234:84 , srv837494862:84 , srv850240421:84 , srv867185234:84 , srv921056893:84 , srv967917557:84 , srv1035116893:85 , srv1084260722:85 , srv1127495963:85 , srv1257678659:85 , srv1411265639:85 , srv1440150290:85 , srv1490233556:85 , srv1492924035:85 , srv1543938853:85 , srv1559461113:85 , srv168956454:85 , srv1738665106:85 , srv1806363812:85 , srv1866989004:85 , srv1878899026:85 , srv1885261242:85 , srv1924113130:85 , srv1936385292:85 , srv1967393235:85 , srv1995618427:85 , srv2007370742:85 , srv2084235363:85 , srv216890186:85 , srv244944334:85 , srv290808437:85 , srv319508232:85 , srv356214082:85 , srv406858178:85 , srv420340824:85 , srv428140188:85 , srv473551831:85 , srv603659469:85 , srv676449597:85 , srv745436037:85 , srv757316437:85 , srv875179863:85 , srv960729498:85 , srv965325346:85 , srv1120854884:86 , srv127600704:86 , srv1442193722:86 , srv1600922175:86 , srv1793395269:86 , srv1854254300:86 , srv1857421453:86 , srv1882816802:86 , srv2005375347:86 , srv2030975885:86 , srv2129999758:86 , srv229845845:86 , srv284504755:86 , srv308292653:86 , srv347715075:86 , srv358555660:86 , srv411577880:86 , srv576058804:86 , srv742721408:86 , srv862361609:86 , srv91011834:86 , srv1580640106:87 , srv1076062989:607 } 2023-08-23 01:53:15,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 01:53:15,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv273700671=59, srv2007370742=51, srv347715075=65, srv428140188=73, srv837494862=89, srv1175824347=8, srv1600922175=27, srv603659469=81, srv1854233040=34, srv1793395269=32, srv229845845=57, srv1019809375=0, srv1854254300=35, srv1967393235=47, srv2005375347=50, srv960729498=97, srv391784551=69, srv1543938853=22, srv290808437=61, srv1092857921=4, srv1329136819=13, srv592877183=80, srv745436037=86, srv1806363812=33, srv473551831=76, srv446971066=74, srv1440150290=16, srv2129999758=55, srv127600704=12, srv1120854884=5, srv757316437=87, srv411577880=71, srv356214082=66, srv2027798578=52, srv1411265639=15, srv1878899026=39, srv1882816802=40, srv1471702152=18, srv406858178=70, srv1944261833=45, srv370776968=68, srv742721408=85, srv1035116893=1, srv420340824=72, srv875179863=93, srv284504755=60, srv850240421=90, srv1885261242=41, srv1559461113=23, srv308292653=63, srv216890186=56, srv45765517=75, srv1936385292=44, srv319508232=64, srv1995618427=49, srv779572094=88, srv1492924035=20, srv664868010=82, srv1924113130=43, srv1490233556=19, srv358555660=67, srv862361609=91, srv1076062989=2, srv1144854551=7, srv1084260722=3, srv533439874=78, srv1127495963=6, srv126266550=11, srv921056893=95, srv1864031301=37, srv1994672933=48, srv91011834=94, srv867185234=92, srv1442193722=17, srv1616581851=28, srv244944334=58, srv1370854964=14, srv1955624000=46, srv2084235363=54, srv306877577=62, srv576058804=79, srv482931484=77, srv958266480=96, srv965325346=98, srv1203898101=9, srv1787327500=31, srv168956454=29, srv2030975885=53, srv676449597=83, srv1584016614=26, srv1857421453=36, srv1738665106=30, srv1519301605=21, srv1901880297=42, srv1866989004=38, srv1257678659=10, srv1580640106=25, srv967917557=99, srv1567442895=24, srv696957234=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 01:53:15,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 01:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 01:53:15,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 01:53:15,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-23 01:53:15,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-23 01:53:15,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-23 01:53:15,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-23 01:53:15,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-23 01:53:15,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-23 01:53:15,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-23 01:53:15,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-23 01:53:15,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-23 01:53:15,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-23 01:53:15,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-23 01:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-23 01:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-23 01:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-23 01:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-23 01:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-23 01:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-23 01:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-23 01:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-23 01:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-23 01:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-23 01:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-23 01:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-23 01:53:15,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-23 01:53:15,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-23 01:53:15,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-23 01:53:15,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-23 01:53:15,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-23 01:53:15,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-23 01:53:15,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-23 01:53:15,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-23 01:53:15,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-23 01:53:15,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-23 01:53:15,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-23 01:53:15,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-23 01:53:15,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-23 01:53:15,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-23 01:53:15,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-23 01:53:15,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-23 01:53:15,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-23 01:53:15,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-23 01:53:15,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-23 01:53:15,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-23 01:53:15,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-23 01:53:15,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-23 01:53:15,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-23 01:53:15,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-23 01:53:15,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-23 01:53:15,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-23 01:53:15,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-23 01:53:15,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-23 01:53:15,539 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:53:15,540 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-23 01:53:15,540 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.944254053974928E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-23 01:53:15,796 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 357 ms to try 5000 different iterations. Found a solution that moves 34 regions; Going from a computed imbalance of 4.944254053974928E-5 to a new imbalance of 7.927494888788932E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.003777777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:53:15,797 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1120854884:83 , srv127600704:83 , srv1519301605:83 , srv1580640106:83 , srv1600922175:83 , srv1793395269:83 , srv1857421453:83 , srv1882816802:83 , srv1995618427:83 , srv216890186:83 , srv284504755:83 , srv742721408:83 , srv1019809375:84 , srv1035116893:84 , srv1084260722:84 , srv1092857921:84 , srv1127495963:84 , srv1175824347:84 , srv1203898101:84 , srv1257678659:84 , srv126266550:84 , srv1329136819:84 , srv1370854964:84 , srv1411265639:84 , srv1440150290:84 , srv1442193722:84 , srv1471702152:84 , srv1490233556:84 , srv1492924035:84 , srv1559461113:84 , srv1584016614:84 , srv1616581851:84 , srv168956454:84 , srv1738665106:84 , srv1787327500:84 , srv1806363812:84 , srv1854254300:84 , srv1864031301:84 , srv1866989004:84 , srv1878899026:84 , srv1885261242:84 , srv1901880297:84 , srv1924113130:84 , srv1936385292:84 , srv1944261833:84 , srv1955624000:84 , srv1967393235:84 , srv1994672933:84 , srv2005375347:84 , srv2007370742:84 , srv2027798578:84 , srv2030975885:84 , srv2084235363:84 , srv2129999758:84 , srv229845845:84 , srv244944334:84 , srv273700671:84 , srv290808437:84 , srv308292653:84 , srv319508232:84 , srv347715075:84 , srv358555660:84 , srv370776968:84 , srv391784551:84 , srv406858178:84 , srv411577880:84 , srv420340824:84 , srv428140188:84 , srv45765517:84 , srv473551831:84 , srv482931484:84 , srv576058804:84 , srv603659469:84 , srv676449597:84 , srv696957234:84 , srv745436037:84 , srv757316437:84 , srv779572094:84 , srv837494862:84 , srv850240421:84 , srv862361609:84 , srv867185234:84 , srv875179863:84 , srv91011834:84 , srv958266480:84 , srv960729498:84 , srv965325346:84 , srv967917557:84 , srv1144854551:85 , srv1543938853:85 , srv1567442895:85 , srv1854233040:85 , srv306877577:85 , srv356214082:85 , srv446971066:85 , srv533439874:85 , srv592877183:85 , srv664868010:85 , srv921056893:85 , srv1076062989:685 } 2023-08-23 01:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 01:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv273700671=59, srv2007370742=51, srv347715075=65, srv428140188=73, srv837494862=89, srv1175824347=8, srv1600922175=27, srv603659469=81, srv1854233040=34, srv1793395269=32, srv229845845=57, srv1019809375=0, srv1854254300=35, srv1967393235=47, srv2005375347=50, srv960729498=97, srv391784551=69, srv1543938853=22, srv290808437=61, srv1092857921=4, srv1329136819=13, srv592877183=80, srv745436037=86, srv1806363812=33, srv473551831=76, srv446971066=74, srv1440150290=16, srv2129999758=55, srv127600704=12, srv1120854884=5, srv757316437=87, srv411577880=71, srv356214082=66, srv2027798578=52, srv1411265639=15, srv1878899026=39, srv1882816802=40, srv1471702152=18, srv406858178=70, srv1944261833=45, srv370776968=68, srv742721408=85, srv1035116893=1, srv420340824=72, srv875179863=93, srv284504755=60, srv850240421=90, srv1885261242=41, srv1559461113=23, srv308292653=63, srv216890186=56, srv45765517=75, srv1936385292=44, srv319508232=64, srv1995618427=49, srv779572094=88, srv1492924035=20, srv664868010=82, srv1924113130=43, srv1490233556=19, srv358555660=67, srv862361609=91, srv1076062989=2, srv1144854551=7, srv1084260722=3, srv533439874=78, srv1127495963=6, srv126266550=11, srv921056893=95, srv1864031301=37, srv1994672933=48, srv91011834=94, srv867185234=92, srv1442193722=17, srv1616581851=28, srv244944334=58, srv1370854964=14, srv1955624000=46, srv2084235363=54, srv306877577=62, srv576058804=79, srv482931484=77, srv958266480=96, srv965325346=98, srv1203898101=9, srv1787327500=31, srv168956454=29, srv2030975885=53, srv676449597=83, srv1584016614=26, srv1857421453=36, srv1738665106=30, srv1519301605=21, srv1901880297=42, srv1866989004=38, srv1257678659=10, srv1580640106=25, srv967917557=99, srv1567442895=24, srv696957234=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 01:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 01:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 01:53:15,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 01:53:15,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-23 01:53:15,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-23 01:53:15,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-23 01:53:15,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-23 01:53:15,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-23 01:53:15,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-23 01:53:15,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-23 01:53:15,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-23 01:53:15,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-23 01:53:15,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-23 01:53:15,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-23 01:53:15,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-23 01:53:15,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-23 01:53:15,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-23 01:53:15,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-23 01:53:15,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-23 01:53:15,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-23 01:53:15,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-23 01:53:15,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-23 01:53:15,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-23 01:53:15,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-23 01:53:15,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-23 01:53:15,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-23 01:53:15,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-23 01:53:15,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-23 01:53:15,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-23 01:53:15,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-23 01:53:15,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-23 01:53:15,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-23 01:53:15,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-23 01:53:15,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-23 01:53:15,939 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:53:15,940 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1120854884:83 , srv127600704:83 , srv1519301605:83 , srv1580640106:83 , srv1600922175:83 , srv1793395269:83 , srv1857421453:83 , srv1882816802:83 , srv1995618427:83 , srv216890186:83 , srv284504755:83 , srv742721408:83 , srv1019809375:84 , srv1035116893:84 , srv1084260722:84 , srv1092857921:84 , srv1127495963:84 , srv1175824347:84 , srv1203898101:84 , srv1257678659:84 , srv126266550:84 , srv1329136819:84 , srv1370854964:84 , srv1411265639:84 , srv1440150290:84 , srv1442193722:84 , srv1471702152:84 , srv1490233556:84 , srv1492924035:84 , srv1559461113:84 , srv1584016614:84 , srv1616581851:84 , srv168956454:84 , srv1738665106:84 , srv1787327500:84 , srv1806363812:84 , srv1854254300:84 , srv1864031301:84 , srv1866989004:84 , srv1878899026:84 , srv1885261242:84 , srv1901880297:84 , srv1924113130:84 , srv1936385292:84 , srv1944261833:84 , srv1955624000:84 , srv1967393235:84 , srv1994672933:84 , srv2005375347:84 , srv2007370742:84 , srv2027798578:84 , srv2030975885:84 , srv2084235363:84 , srv2129999758:84 , srv229845845:84 , srv244944334:84 , srv273700671:84 , srv290808437:84 , srv308292653:84 , srv319508232:84 , srv347715075:84 , srv358555660:84 , srv370776968:84 , srv391784551:84 , srv406858178:84 , srv411577880:84 , srv420340824:84 , srv428140188:84 , srv45765517:84 , srv473551831:84 , srv482931484:84 , srv576058804:84 , srv603659469:84 , srv676449597:84 , srv696957234:84 , srv745436037:84 , srv757316437:84 , srv779572094:84 , srv837494862:84 , srv850240421:84 , srv862361609:84 , srv867185234:84 , srv875179863:84 , srv91011834:84 , srv958266480:84 , srv960729498:84 , srv965325346:84 , srv967917557:84 , srv1144854551:85 , srv1543938853:85 , srv1567442895:85 , srv1854233040:85 , srv306877577:85 , srv356214082:85 , srv446971066:85 , srv533439874:85 , srv592877183:85 , srv664868010:85 , srv921056893:85 , srv1076062989:685 } 2023-08-23 01:53:15,970 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=236 (was 213) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=11530 (was 12366) 2023-08-23 01:53:15,980 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=236, ProcessCount=171, AvailableMemoryMB=11530 2023-08-23 01:53:15,981 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-23 01:53:15,981 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-23 01:53:15,991 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1031135919:84 , srv1074488467:84 , srv1298359033:84 , srv1312200020:84 , srv1353064853:84 , srv1488115964:84 , srv1552227705:84 , srv1557778213:84 , srv1567122103:84 , srv1578516580:84 , srv1607606576:84 , srv1694447620:84 , srv1733020332:84 , srv1745617828:84 , srv1787595727:84 , srv196903374:84 , srv2102941441:84 , srv260841496:84 , srv321941451:84 , srv382173751:84 , srv443151916:84 , srv472125007:84 , srv621325105:84 , srv689940336:84 , srv816139969:84 , srv915992693:84 , srv937834595:84 , srv969172680:84 , srv983083032:84 , srv356753853:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-23 01:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 01:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1488115964=5, srv1552227705=6, srv621325105=23, srv1312200020=3, srv196903374=15, srv2102941441=16, srv1607606576=10, srv1298359033=2, srv689940336=24, srv472125007=22, srv321941451=18, srv983083032=29, srv443151916=21, srv1031135919=0, srv1557778213=7, srv1353064853=4, srv1578516580=9, srv937834595=27, srv260841496=17, srv1074488467=1, srv816139969=25, srv382173751=20, srv1694447620=11, srv1787595727=14, srv356753853=19, srv1567122103=8, srv915992693=26, srv1745617828=13, srv969172680=28, srv1733020332=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 01:53:16,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 01:53:16,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-23 01:53:16,030 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.150951265806541, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:53:16,031 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912668685773823, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.150951265806541, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-23 01:55:16,010 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6141112 different iterations. Found a solution that moves 1851 regions; Going from a computed imbalance of 0.9912668685773823 to a new imbalance of 4.320326340186438E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:55:16,018 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1031135919:90 , srv1074488467:90 , srv1298359033:90 , srv1312200020:90 , srv1353064853:90 , srv1488115964:90 , srv1552227705:90 , srv1557778213:90 , srv1567122103:90 , srv1578516580:90 , srv1607606576:90 , srv1694447620:90 , srv1733020332:90 , srv1745617828:90 , srv1787595727:90 , srv196903374:90 , srv2102941441:90 , srv260841496:90 , srv321941451:90 , srv356753853:90 , srv382173751:90 , srv443151916:90 , srv472125007:90 , srv621325105:90 , srv689940336:90 , srv816139969:90 , srv915992693:90 , srv937834595:90 , srv969172680:90 , srv983083032:90 } 2023-08-23 01:55:16,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-23 01:55:16,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1488115964=5, srv1552227705=6, srv621325105=23, srv1312200020=3, srv196903374=15, srv2102941441=16, srv1607606576=10, srv1298359033=2, srv689940336=24, srv472125007=22, srv321941451=18, srv983083032=29, srv443151916=21, srv1031135919=0, srv1557778213=7, srv1353064853=4, srv1578516580=9, srv937834595=27, srv260841496=17, srv1074488467=1, srv816139969=25, srv382173751=20, srv1694447620=11, srv1787595727=14, srv356753853=19, srv1567122103=8, srv915992693=26, srv1745617828=13, srv969172680=28, srv1733020332=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-23 01:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-23 01:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-23 01:55:16,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-23 01:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-23 01:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-23 01:55:16,041 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-23 01:55:16,041 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1031135919:90 , srv1074488467:90 , srv1298359033:90 , srv1312200020:90 , srv1353064853:90 , srv1488115964:90 , srv1552227705:90 , srv1557778213:90 , srv1567122103:90 , srv1578516580:90 , srv1607606576:90 , srv1694447620:90 , srv1733020332:90 , srv1745617828:90 , srv1787595727:90 , srv196903374:90 , srv2102941441:90 , srv260841496:90 , srv321941451:90 , srv356753853:90 , srv382173751:90 , srv443151916:90 , srv472125007:90 , srv621325105:90 , srv689940336:90 , srv816139969:90 , srv915992693:90 , srv937834595:90 , srv969172680:90 , srv983083032:90 } 2023-08-23 01:55:16,054 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=200 (was 236), ProcessCount=169 (was 171), AvailableMemoryMB=11671 (was 11530) - AvailableMemoryMB LEAK? -