2024-05-03 02:01:10,445 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-05-03 02:01:10,710 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-05-03 02:01:10,809 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-05-03 02:01:10,870 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-05-03 02:01:10,878 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-05-03 02:01:10,893 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=231, ProcessCount=175, AvailableMemoryMB=2588 2024-05-03 02:01:10,895 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-05-03 02:01:10,896 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-05-03 02:01:11,030 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1027895821:84 , srv1045353445:84 , srv1052543222:84 , srv1064898369:84 , srv1084027294:84 , srv1119491812:84 , srv1120450366:84 , srv118163806:84 , srv1186183717:84 , srv1188096529:84 , srv122403027:84 , srv1235885708:84 , srv123630834:84 , srv1236977944:84 , srv1241283291:84 , srv1242911035:84 , srv1253879304:84 , srv1255003997:84 , srv1281307152:84 , srv1314850117:84 , srv1328359830:84 , srv1330557898:84 , srv1330871135:84 , srv1350736448:84 , srv1362087693:84 , srv1401279790:84 , srv1417777656:84 , srv1427690977:84 , srv1438011461:84 , srv1439341820:84 , srv1441587821:84 , srv1446747859:84 , srv1456258571:84 , srv1494859801:84 , srv1533418984:84 , srv1548148263:84 , srv1593548918:84 , srv1608408345:84 , srv1625345084:84 , srv1690735239:84 , srv1728833013:84 , srv1733819850:84 , srv1771913450:84 , srv1807850272:84 , srv1813477927:84 , srv1843855291:84 , srv1861609016:84 , srv1878977787:84 , srv1887501821:84 , srv1915227060:84 , srv1929843505:84 , srv1949057542:84 , srv1950770020:84 , srv1964445756:84 , srv196673041:84 , srv1979048046:84 , srv2042028071:84 , srv2070018910:84 , srv2076113396:84 , srv2080638778:84 , srv209338870:84 , srv2133521317:84 , srv2135153034:84 , srv2139930460:84 , srv218667123:84 , srv222215344:84 , srv236686949:84 , srv266181425:84 , srv266779870:84 , srv276298898:84 , srv277438289:84 , srv311665603:84 , srv326102508:84 , srv358366243:84 , srv363078329:84 , srv41561453:84 , srv42824822:84 , srv462289127:84 , srv498613265:84 , srv499592528:84 , srv56071373:84 , srv597020934:84 , srv660813612:84 , srv674177631:84 , srv701771231:84 , srv730172652:84 , srv731901515:84 , srv735352436:84 , srv755664355:84 , srv761697121:84 , srv773406401:84 , srv785455784:84 , srv796262840:84 , srv802327564:84 , srv816435379:84 , srv876826478:84 , srv891639221:84 , srv916422822:84 , srv925895287:84 , srv896951059:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-05-03 02:01:11,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-05-03 02:01:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1401279790=25, srv2135153034=62, srv876826478=95, srv1733819850=41, srv1241283291=14, srv674177631=83, srv1979048046=55, srv236686949=66, srv802327564=93, srv1843855291=45, srv266779870=68, srv1949057542=51, srv1186183717=8, srv1314850117=19, srv1690735239=39, srv462289127=77, srv1929843505=50, srv277438289=70, srv1771913450=42, srv925895287=99, srv1728833013=40, srv2139930460=63, srv1330871135=22, srv730172652=85, srv891639221=96, srv122403027=10, srv123630834=12, srv2070018910=57, srv498613265=78, srv2042028071=56, srv816435379=94, srv1417777656=26, srv1964445756=53, srv1064898369=3, srv1255003997=17, srv1456258571=32, srv1328359830=20, srv796262840=92, srv735352436=87, srv761697121=89, srv222215344=65, srv2076113396=58, srv1807850272=43, srv773406401=90, srv660813612=82, srv1253879304=16, srv1188096529=9, srv118163806=7, srv1608408345=37, srv755664355=88, srv2133521317=61, srv1052543222=2, srv1120450366=6, srv1235885708=11, srv1281307152=18, srv1084027294=4, srv1446747859=31, srv1915227060=49, srv326102508=72, srv1887501821=48, srv1625345084=38, srv1813477927=44, srv41561453=75, srv266181425=67, srv785455784=91, srv358366243=73, srv363078329=74, srv1533418984=34, srv42824822=76, srv1045353445=1, srv1950770020=52, srv1439341820=29, srv1242911035=15, srv2080638778=59, srv1494859801=33, srv196673041=54, srv731901515=86, srv1350736448=23, srv1119491812=5, srv1861609016=46, srv209338870=60, srv218667123=64, srv1330557898=21, srv1438011461=28, srv499592528=79, srv1441587821=30, srv1427690977=27, srv311665603=71, srv1236977944=13, srv597020934=81, srv56071373=80, srv896951059=97, srv916422822=98, srv1548148263=35, srv701771231=84, srv1362087693=24, srv1878977787=47, srv1593548918=36, srv1027895821=0, srv276298898=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-05-03 02:01:11,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-05-03 02:01:11,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-05-03 02:01:11,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-05-03 02:01:11,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-05-03 02:01:11,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-05-03 02:01:11,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-05-03 02:01:11,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-05-03 02:01:11,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-05-03 02:01:11,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-05-03 02:01:11,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-05-03 02:01:11,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-05-03 02:01:11,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-05-03 02:01:11,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-05-03 02:01:11,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-05-03 02:01:11,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-05-03 02:01:11,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-05-03 02:01:11,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-05-03 02:01:11,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-05-03 02:01:11,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-05-03 02:01:11,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-05-03 02:01:11,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-05-03 02:01:11,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-05-03 02:01:11,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-05-03 02:01:11,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-05-03 02:01:11,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-05-03 02:01:11,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-05-03 02:01:11,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-05-03 02:01:11,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-05-03 02:01:11,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-05-03 02:01:11,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-05-03 02:01:11,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-05-03 02:01:11,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-05-03 02:01:11,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-05-03 02:01:11,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-05-03 02:01:11,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-05-03 02:01:11,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-05-03 02:01:11,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-05-03 02:01:11,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-05-03 02:01:11,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-05-03 02:01:11,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-05-03 02:01:11,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-05-03 02:01:11,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-05-03 02:01:11,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-05-03 02:01:11,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-05-03 02:01:11,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-05-03 02:01:11,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-05-03 02:01:11,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-05-03 02:01:11,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-05-03 02:01:11,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-05-03 02:01:11,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-05-03 02:01:11,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-05-03 02:01:11,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-05-03 02:01:11,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-05-03 02:01:11,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-05-03 02:01:11,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-05-03 02:01:11,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-05-03 02:01:11,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-05-03 02:01:11,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-05-03 02:01:11,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-05-03 02:01:11,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-05-03 02:01:11,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-05-03 02:01:11,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-05-03 02:01:11,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-05-03 02:01:11,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-05-03 02:01:11,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-05-03 02:01:11,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-05-03 02:01:11,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-05-03 02:01:11,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-05-03 02:01:11,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-05-03 02:01:11,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-05-03 02:01:11,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-05-03 02:01:11,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-05-03 02:01:11,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-05-03 02:01:11,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-05-03 02:01:11,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-05-03 02:01:11,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-05-03 02:01:11,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-05-03 02:01:11,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-05-03 02:01:11,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-05-03 02:01:11,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-05-03 02:01:11,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-05-03 02:01:11,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-05-03 02:01:11,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-05-03 02:01:11,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-05-03 02:01:11,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-05-03 02:01:11,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-05-03 02:01:11,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-05-03 02:01:11,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-05-03 02:01:11,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-05-03 02:01:11,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-05-03 02:01:11,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-05-03 02:01:11,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-05-03 02:01:11,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-05-03 02:01:11,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-05-03 02:01:11,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-05-03 02:01:11,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-05-03 02:01:11,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-05-03 02:01:11,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-05-03 02:01:11,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-05-03 02:01:11,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-05-03 02:01:11,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-05-03 02:01:11,341 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:01:11,341 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-05-03 02:01:11,341 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-05-03 02:01:11,672 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 472 ms to try 5000 different iterations. Found a solution that moves 4680 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1759741201098826. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.042382882944405, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05572499414034637, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.52, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.042382882944405, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17141666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23508333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:01:11,681 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv56071373:81 , srv123630834:82 , srv1241283291:82 , srv277438289:82 , srv701771231:82 , srv1027895821:83 , srv1362087693:83 , srv1456258571:83 , srv1494859801:83 , srv363078329:83 , srv755664355:83 , srv876826478:83 , srv1441587821:84 , srv1548148263:84 , srv1728833013:84 , srv1733819850:84 , srv1807850272:84 , srv1843855291:84 , srv1949057542:84 , srv236686949:84 , srv499592528:84 , srv802327564:84 , srv118163806:85 , srv1186183717:85 , srv122403027:85 , srv1235885708:85 , srv1281307152:85 , srv1350736448:85 , srv1533418984:85 , srv1861609016:85 , srv2042028071:85 , srv2076113396:85 , srv218667123:85 , srv326102508:85 , srv597020934:85 , srv816435379:85 , srv891639221:85 , srv1064898369:86 , srv1084027294:86 , srv1236977944:86 , srv1253879304:86 , srv1330871135:86 , srv1417777656:86 , srv1438011461:86 , srv1446747859:86 , srv1690735239:86 , srv1878977787:86 , srv1887501821:86 , srv209338870:86 , srv358366243:86 , srv42824822:86 , srv462289127:86 , srv916422822:86 , srv925895287:86 , srv1052543222:87 , srv1242911035:87 , srv1314850117:87 , srv1330557898:87 , srv1401279790:87 , srv1427690977:87 , srv1608408345:87 , srv1625345084:87 , srv1771913450:87 , srv1915227060:87 , srv1950770020:87 , srv1964445756:87 , srv2133521317:87 , srv2135153034:87 , srv266181425:87 , srv276298898:87 , srv773406401:87 , srv785455784:87 , srv1439341820:88 , srv1929843505:88 , srv196673041:88 , srv1979048046:88 , srv2070018910:88 , srv2080638778:88 , srv2139930460:88 , srv222215344:88 , srv266779870:88 , srv498613265:88 , srv730172652:88 , srv735352436:88 , srv796262840:88 , srv1119491812:89 , srv1188096529:89 , srv1255003997:89 , srv1328359830:89 , srv311665603:89 , srv731901515:89 , srv761697121:89 , srv1045353445:90 , srv1120450366:90 , srv1593548918:90 , srv1813477927:90 , srv660813612:90 , srv674177631:90 , srv41561453:93 , srv896951059:467 } 2024-05-03 02:01:11,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-05-03 02:01:11,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1401279790=25, srv2135153034=62, srv876826478=95, srv1733819850=41, srv1241283291=14, srv674177631=83, srv1979048046=55, srv236686949=66, srv802327564=93, srv1843855291=45, srv266779870=68, srv1949057542=51, srv1186183717=8, srv1314850117=19, srv1690735239=39, srv462289127=77, srv1929843505=50, srv277438289=70, srv1771913450=42, srv925895287=99, srv1728833013=40, srv2139930460=63, srv1330871135=22, srv730172652=85, srv891639221=96, srv122403027=10, srv123630834=12, srv2070018910=57, srv498613265=78, srv2042028071=56, srv816435379=94, srv1417777656=26, srv1964445756=53, srv1064898369=3, srv1255003997=17, srv1456258571=32, srv1328359830=20, srv796262840=92, srv735352436=87, srv761697121=89, srv222215344=65, srv2076113396=58, srv1807850272=43, srv773406401=90, srv660813612=82, srv1253879304=16, srv1188096529=9, srv118163806=7, srv1608408345=37, srv755664355=88, srv2133521317=61, srv1052543222=2, srv1120450366=6, srv1235885708=11, srv1281307152=18, srv1084027294=4, srv1446747859=31, srv1915227060=49, srv326102508=72, srv1887501821=48, srv1625345084=38, srv1813477927=44, srv41561453=75, srv266181425=67, srv785455784=91, srv358366243=73, srv363078329=74, srv1533418984=34, srv42824822=76, srv1045353445=1, srv1950770020=52, srv1439341820=29, srv1242911035=15, srv2080638778=59, srv1494859801=33, srv196673041=54, srv731901515=86, srv1350736448=23, srv1119491812=5, srv1861609016=46, srv209338870=60, srv218667123=64, srv1330557898=21, srv1438011461=28, srv499592528=79, srv1441587821=30, srv1427690977=27, srv311665603=71, srv1236977944=13, srv597020934=81, srv56071373=80, srv896951059=97, srv916422822=98, srv1548148263=35, srv701771231=84, srv1362087693=24, srv1878977787=47, srv1593548918=36, srv1027895821=0, srv276298898=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-05-03 02:01:11,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-05-03 02:01:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-05-03 02:01:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-05-03 02:01:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-05-03 02:01:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-05-03 02:01:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-05-03 02:01:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-05-03 02:01:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-05-03 02:01:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-05-03 02:01:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-05-03 02:01:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-05-03 02:01:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-05-03 02:01:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-05-03 02:01:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-05-03 02:01:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-05-03 02:01:11,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-05-03 02:01:11,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-05-03 02:01:11,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-05-03 02:01:11,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-05-03 02:01:11,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-05-03 02:01:11,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-05-03 02:01:11,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-05-03 02:01:11,803 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.042382882944405, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05572499414034637, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.042382882944405, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17141666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23508333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:01:11,803 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-05-03 02:01:11,803 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17594135004947406, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.042382882944405, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05572499414034637, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.042382882944405, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17141666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23508333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-05-03 02:01:11,972 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 250 ms to try 5000 different iterations. Found a solution that moves 2591 regions; Going from a computed imbalance of 0.17594135004947406 to a new imbalance of 0.016106183350109785. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023397645324029208); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03732046435246655, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2878888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023397645324029208); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.048916666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:01:11,979 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv41561453:78 , srv1045353445:80 , srv1120450366:82 , srv1188096529:82 , srv1255003997:82 , srv1593548918:82 , srv1813477927:82 , srv498613265:82 , srv761697121:82 , srv1439341820:83 , srv1608408345:83 , srv1771913450:83 , srv1929843505:83 , srv2070018910:83 , srv222215344:83 , srv311665603:83 , srv660813612:83 , srv730172652:83 , srv731901515:83 , srv785455784:83 , srv796262840:83 , srv1236977944:84 , srv1314850117:84 , srv1328359830:84 , srv1401279790:84 , srv1878977787:84 , srv1915227060:84 , srv1950770020:84 , srv2076113396:84 , srv2139930460:84 , srv1064898369:85 , srv1119491812:85 , srv122403027:85 , srv1242911035:85 , srv1417777656:85 , srv1438011461:85 , srv1625345084:85 , srv1690735239:85 , srv1861609016:85 , srv1964445756:85 , srv196673041:85 , srv2080638778:85 , srv2133521317:85 , srv2135153034:85 , srv266181425:85 , srv266779870:85 , srv42824822:85 , srv735352436:85 , srv773406401:85 , srv891639221:85 , srv1084027294:86 , srv1186183717:86 , srv1235885708:86 , srv1253879304:86 , srv1330871135:86 , srv1446747859:86 , srv2042028071:86 , srv209338870:86 , srv236686949:86 , srv276298898:86 , srv816435379:86 , srv1052543222:87 , srv118163806:87 , srv1281307152:87 , srv1330557898:87 , srv1350736448:87 , srv1362087693:87 , srv1427690977:87 , srv1533418984:87 , srv1733819850:87 , srv1887501821:87 , srv1949057542:87 , srv218667123:87 , srv326102508:87 , srv462289127:87 , srv597020934:87 , srv925895287:87 , srv1441587821:88 , srv1843855291:88 , srv1979048046:88 , srv363078329:88 , srv674177631:88 , srv916422822:88 , srv1027895821:89 , srv1241283291:89 , srv1456258571:89 , srv1548148263:89 , srv1728833013:89 , srv1807850272:89 , srv277438289:89 , srv358366243:89 , srv755664355:89 , srv802327564:89 , srv876826478:89 , srv123630834:90 , srv499592528:91 , srv701771231:91 , srv1494859801:92 , srv56071373:92 , srv896951059:515 } 2024-05-03 02:01:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-05-03 02:01:11,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1401279790=25, srv2135153034=62, srv876826478=95, srv1733819850=41, srv1241283291=14, srv674177631=83, srv1979048046=55, srv236686949=66, srv802327564=93, srv1843855291=45, srv266779870=68, srv1949057542=51, srv1186183717=8, srv1314850117=19, srv1690735239=39, srv462289127=77, srv1929843505=50, srv277438289=70, srv1771913450=42, srv925895287=99, srv1728833013=40, srv2139930460=63, srv1330871135=22, srv730172652=85, srv891639221=96, srv122403027=10, srv123630834=12, srv2070018910=57, srv498613265=78, srv2042028071=56, srv816435379=94, srv1417777656=26, srv1964445756=53, srv1064898369=3, srv1255003997=17, srv1456258571=32, srv1328359830=20, srv796262840=92, srv735352436=87, srv761697121=89, srv222215344=65, srv2076113396=58, srv1807850272=43, srv773406401=90, srv660813612=82, srv1253879304=16, srv1188096529=9, srv118163806=7, srv1608408345=37, srv755664355=88, srv2133521317=61, srv1052543222=2, srv1120450366=6, srv1235885708=11, srv1281307152=18, srv1084027294=4, srv1446747859=31, srv1915227060=49, srv326102508=72, srv1887501821=48, srv1625345084=38, srv1813477927=44, srv41561453=75, srv266181425=67, srv785455784=91, srv358366243=73, srv363078329=74, srv1533418984=34, srv42824822=76, srv1045353445=1, srv1950770020=52, srv1439341820=29, srv1242911035=15, srv2080638778=59, srv1494859801=33, srv196673041=54, srv731901515=86, srv1350736448=23, srv1119491812=5, srv1861609016=46, srv209338870=60, srv218667123=64, srv1330557898=21, srv1438011461=28, srv499592528=79, srv1441587821=30, srv1427690977=27, srv311665603=71, srv1236977944=13, srv597020934=81, srv56071373=80, srv896951059=97, srv916422822=98, srv1548148263=35, srv701771231=84, srv1362087693=24, srv1878977787=47, srv1593548918=36, srv1027895821=0, srv276298898=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-05-03 02:01:11,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-05-03 02:01:11,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-05-03 02:01:11,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-05-03 02:01:11,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-05-03 02:01:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-05-03 02:01:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-05-03 02:01:12,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-05-03 02:01:12,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-05-03 02:01:12,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-05-03 02:01:12,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-05-03 02:01:12,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-05-03 02:01:12,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-05-03 02:01:12,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-05-03 02:01:12,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-05-03 02:01:12,089 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023397645324029208); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03732046435246655, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023397645324029208); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.048916666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:01:12,089 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-05-03 02:01:12,089 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016088040780340866, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023397645324029208); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03732046435246655, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023397645324029208); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.048916666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-05-03 02:01:12,241 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 226 ms to try 5000 different iterations. Found a solution that moves 1072 regions; Going from a computed imbalance of 0.016088040780340866 to a new imbalance of 9.222320637816613E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006007436053853573); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008448696430838983); MoveCostFunction : (multiplier=7.0, imbalance=0.11911111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006007436053853573); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:01:12,242 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv674177631:79 , srv358366243:82 , srv1330557898:83 , srv1494859801:83 , srv1979048046:83 , srv916422822:83 , srv925895287:83 , srv1119491812:84 , srv1427690977:84 , srv1441587821:84 , srv1728833013:84 , srv266779870:84 , srv276298898:84 , srv462289127:84 , srv499592528:84 , srv56071373:84 , srv660813612:84 , srv1027895821:85 , srv1084027294:85 , srv118163806:85 , srv1186183717:85 , srv1253879304:85 , srv1281307152:85 , srv1330871135:85 , srv1350736448:85 , srv1446747859:85 , srv1456258571:85 , srv1548148263:85 , srv1807850272:85 , srv1813477927:85 , srv1887501821:85 , srv1964445756:85 , srv196673041:85 , srv2070018910:85 , srv2080638778:85 , srv2133521317:85 , srv2135153034:85 , srv222215344:85 , srv266181425:85 , srv311665603:85 , srv597020934:85 , srv701771231:85 , srv735352436:85 , srv755664355:85 , srv773406401:85 , srv802327564:85 , srv1052543222:86 , srv1064898369:86 , srv1120450366:86 , srv1188096529:86 , srv123630834:86 , srv1236977944:86 , srv1241283291:86 , srv1242911035:86 , srv1255003997:86 , srv1328359830:86 , srv1438011461:86 , srv1439341820:86 , srv1533418984:86 , srv1593548918:86 , srv1608408345:86 , srv1625345084:86 , srv1843855291:86 , srv1929843505:86 , srv1950770020:86 , srv2042028071:86 , srv2139930460:86 , srv218667123:86 , srv326102508:86 , srv41561453:86 , srv730172652:86 , srv731901515:86 , srv761697121:86 , srv816435379:86 , srv876826478:86 , srv1045353445:87 , srv1235885708:87 , srv1401279790:87 , srv1417777656:87 , srv1690735239:87 , srv1733819850:87 , srv1771913450:87 , srv1861609016:87 , srv1915227060:87 , srv209338870:87 , srv236686949:87 , srv277438289:87 , srv363078329:87 , srv42824822:87 , srv785455784:87 , srv796262840:87 , srv891639221:87 , srv122403027:88 , srv1314850117:88 , srv1362087693:88 , srv1878977787:88 , srv1949057542:88 , srv498613265:88 , srv2076113396:89 , srv896951059:529 } 2024-05-03 02:01:12,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-05-03 02:01:12,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1401279790=25, srv2135153034=62, srv876826478=95, srv1733819850=41, srv1241283291=14, srv674177631=83, srv1979048046=55, srv236686949=66, srv802327564=93, srv1843855291=45, srv266779870=68, srv1949057542=51, srv1186183717=8, srv1314850117=19, srv1690735239=39, srv462289127=77, srv1929843505=50, srv277438289=70, srv1771913450=42, srv925895287=99, srv1728833013=40, srv2139930460=63, srv1330871135=22, srv730172652=85, srv891639221=96, srv122403027=10, srv123630834=12, srv2070018910=57, srv498613265=78, srv2042028071=56, srv816435379=94, srv1417777656=26, srv1964445756=53, srv1064898369=3, srv1255003997=17, srv1456258571=32, srv1328359830=20, srv796262840=92, srv735352436=87, srv761697121=89, srv222215344=65, srv2076113396=58, srv1807850272=43, srv773406401=90, srv660813612=82, srv1253879304=16, srv1188096529=9, srv118163806=7, srv1608408345=37, srv755664355=88, srv2133521317=61, srv1052543222=2, srv1120450366=6, srv1235885708=11, srv1281307152=18, srv1084027294=4, srv1446747859=31, srv1915227060=49, srv326102508=72, srv1887501821=48, srv1625345084=38, srv1813477927=44, srv41561453=75, srv266181425=67, srv785455784=91, srv358366243=73, srv363078329=74, srv1533418984=34, srv42824822=76, srv1045353445=1, srv1950770020=52, srv1439341820=29, srv1242911035=15, srv2080638778=59, srv1494859801=33, srv196673041=54, srv731901515=86, srv1350736448=23, srv1119491812=5, srv1861609016=46, srv209338870=60, srv218667123=64, srv1330557898=21, srv1438011461=28, srv499592528=79, srv1441587821=30, srv1427690977=27, srv311665603=71, srv1236977944=13, srv597020934=81, srv56071373=80, srv896951059=97, srv916422822=98, srv1548148263=35, srv701771231=84, srv1362087693=24, srv1878977787=47, srv1593548918=36, srv1027895821=0, srv276298898=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-05-03 02:01:12,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-05-03 02:01:12,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-05-03 02:01:12,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-05-03 02:01:12,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-05-03 02:01:12,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-05-03 02:01:12,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-05-03 02:01:12,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-05-03 02:01:12,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-05-03 02:01:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-05-03 02:01:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-05-03 02:01:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-05-03 02:01:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-05-03 02:01:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-05-03 02:01:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-05-03 02:01:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-05-03 02:01:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-05-03 02:01:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-05-03 02:01:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-05-03 02:01:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-05-03 02:01:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-05-03 02:01:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-05-03 02:01:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-05-03 02:01:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-05-03 02:01:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-05-03 02:01:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-05-03 02:01:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-05-03 02:01:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-05-03 02:01:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-05-03 02:01:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-05-03 02:01:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-05-03 02:01:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-05-03 02:01:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-05-03 02:01:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-05-03 02:01:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-05-03 02:01:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-05-03 02:01:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-05-03 02:01:12,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-05-03 02:01:12,356 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006007436053853573); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008448696430838983); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006007436053853573); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:01:12,356 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-05-03 02:01:12,356 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.147257593461998E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006007436053853573); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008448696430838983); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006007436053853573); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-05-03 02:01:12,576 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 306 ms to try 5000 different iterations. Found a solution that moves 298 regions; Going from a computed imbalance of 9.147257593461998E-4 to a new imbalance of 6.06099112798128E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.03311111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:01:12,577 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1052543222:83 , srv1119491812:83 , srv1314850117:83 , srv1979048046:83 , srv209338870:83 , srv499592528:83 , srv1027895821:84 , srv1084027294:84 , srv1188096529:84 , srv122403027:84 , srv1235885708:84 , srv123630834:84 , srv1242911035:84 , srv1328359830:84 , srv1362087693:84 , srv1401279790:84 , srv1417777656:84 , srv1427690977:84 , srv1494859801:84 , srv1548148263:84 , srv1593548918:84 , srv1625345084:84 , srv1690735239:84 , srv1733819850:84 , srv1807850272:84 , srv1843855291:84 , srv1878977787:84 , srv1887501821:84 , srv1915227060:84 , srv1949057542:84 , srv1964445756:84 , srv196673041:84 , srv2076113396:84 , srv2080638778:84 , srv2139930460:84 , srv218667123:84 , srv277438289:84 , srv326102508:84 , srv358366243:84 , srv363078329:84 , srv42824822:84 , srv498613265:84 , srv701771231:84 , srv731901515:84 , srv761697121:84 , srv802327564:84 , srv1045353445:85 , srv1064898369:85 , srv1120450366:85 , srv118163806:85 , srv1236977944:85 , srv1241283291:85 , srv1253879304:85 , srv1255003997:85 , srv1281307152:85 , srv1330557898:85 , srv1330871135:85 , srv1350736448:85 , srv1438011461:85 , srv1439341820:85 , srv1456258571:85 , srv1533418984:85 , srv1608408345:85 , srv1728833013:85 , srv1771913450:85 , srv1813477927:85 , srv1861609016:85 , srv1929843505:85 , srv2042028071:85 , srv2133521317:85 , srv2135153034:85 , srv222215344:85 , srv236686949:85 , srv266181425:85 , srv266779870:85 , srv276298898:85 , srv311665603:85 , srv41561453:85 , srv462289127:85 , srv56071373:85 , srv597020934:85 , srv660813612:85 , srv674177631:85 , srv730172652:85 , srv735352436:85 , srv773406401:85 , srv785455784:85 , srv796262840:85 , srv816435379:85 , srv876826478:85 , srv891639221:85 , srv916422822:85 , srv925895287:85 , srv1186183717:86 , srv1446747859:86 , srv1950770020:86 , srv2070018910:86 , srv755664355:86 , srv1441587821:87 , srv896951059:630 } 2024-05-03 02:01:12,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-05-03 02:01:12,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1401279790=25, srv2135153034=62, srv876826478=95, srv1733819850=41, srv1241283291=14, srv674177631=83, srv1979048046=55, srv236686949=66, srv802327564=93, srv1843855291=45, srv266779870=68, srv1949057542=51, srv1186183717=8, srv1314850117=19, srv1690735239=39, srv462289127=77, srv1929843505=50, srv277438289=70, srv1771913450=42, srv925895287=99, srv1728833013=40, srv2139930460=63, srv1330871135=22, srv730172652=85, srv891639221=96, srv122403027=10, srv123630834=12, srv2070018910=57, srv498613265=78, srv2042028071=56, srv816435379=94, srv1417777656=26, srv1964445756=53, srv1064898369=3, srv1255003997=17, srv1456258571=32, srv1328359830=20, srv796262840=92, srv735352436=87, srv761697121=89, srv222215344=65, srv2076113396=58, srv1807850272=43, srv773406401=90, srv660813612=82, srv1253879304=16, srv1188096529=9, srv118163806=7, srv1608408345=37, srv755664355=88, srv2133521317=61, srv1052543222=2, srv1120450366=6, srv1235885708=11, srv1281307152=18, srv1084027294=4, srv1446747859=31, srv1915227060=49, srv326102508=72, srv1887501821=48, srv1625345084=38, srv1813477927=44, srv41561453=75, srv266181425=67, srv785455784=91, srv358366243=73, srv363078329=74, srv1533418984=34, srv42824822=76, srv1045353445=1, srv1950770020=52, srv1439341820=29, srv1242911035=15, srv2080638778=59, srv1494859801=33, srv196673041=54, srv731901515=86, srv1350736448=23, srv1119491812=5, srv1861609016=46, srv209338870=60, srv218667123=64, srv1330557898=21, srv1438011461=28, srv499592528=79, srv1441587821=30, srv1427690977=27, srv311665603=71, srv1236977944=13, srv597020934=81, srv56071373=80, srv896951059=97, srv916422822=98, srv1548148263=35, srv701771231=84, srv1362087693=24, srv1878977787=47, srv1593548918=36, srv1027895821=0, srv276298898=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-05-03 02:01:12,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-05-03 02:01:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-05-03 02:01:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-05-03 02:01:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-05-03 02:01:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-05-03 02:01:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-05-03 02:01:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-05-03 02:01:12,675 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:01:12,676 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-05-03 02:01:12,676 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.852327068114902E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-05-03 02:01:12,828 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 225 ms to try 5000 different iterations. Found a solution that moves 47 regions; Going from a computed imbalance of 5.852327068114902E-5 to a new imbalance of 8.30879115105803E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.005222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:01:12,828 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1052543222:83 , srv1120450366:83 , srv1328359830:83 , srv1441587821:83 , srv1446747859:83 , srv1533418984:83 , srv1949057542:83 , srv1950770020:83 , srv735352436:83 , srv755664355:83 , srv796262840:83 , srv876826478:83 , srv1045353445:84 , srv1064898369:84 , srv118163806:84 , srv1186183717:84 , srv122403027:84 , srv1235885708:84 , srv123630834:84 , srv1241283291:84 , srv1242911035:84 , srv1253879304:84 , srv1255003997:84 , srv1281307152:84 , srv1314850117:84 , srv1330557898:84 , srv1330871135:84 , srv1350736448:84 , srv1362087693:84 , srv1401279790:84 , srv1417777656:84 , srv1427690977:84 , srv1438011461:84 , srv1439341820:84 , srv1456258571:84 , srv1494859801:84 , srv1548148263:84 , srv1593548918:84 , srv1625345084:84 , srv1690735239:84 , srv1728833013:84 , srv1733819850:84 , srv1771913450:84 , srv1807850272:84 , srv1813477927:84 , srv1843855291:84 , srv1861609016:84 , srv1878977787:84 , srv1887501821:84 , srv1915227060:84 , srv1929843505:84 , srv196673041:84 , srv1979048046:84 , srv2042028071:84 , srv2070018910:84 , srv2076113396:84 , srv2080638778:84 , srv209338870:84 , srv2133521317:84 , srv2135153034:84 , srv2139930460:84 , srv218667123:84 , srv236686949:84 , srv266181425:84 , srv266779870:84 , srv276298898:84 , srv277438289:84 , srv311665603:84 , srv326102508:84 , srv363078329:84 , srv41561453:84 , srv42824822:84 , srv462289127:84 , srv498613265:84 , srv499592528:84 , srv56071373:84 , srv597020934:84 , srv660813612:84 , srv674177631:84 , srv701771231:84 , srv730172652:84 , srv731901515:84 , srv773406401:84 , srv785455784:84 , srv802327564:84 , srv816435379:84 , srv891639221:84 , srv916422822:84 , srv1027895821:85 , srv1084027294:85 , srv1119491812:85 , srv1188096529:85 , srv1236977944:85 , srv1608408345:85 , srv1964445756:85 , srv222215344:85 , srv358366243:85 , srv761697121:85 , srv925895287:85 , srv896951059:685 } 2024-05-03 02:01:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-05-03 02:01:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1401279790=25, srv2135153034=62, srv876826478=95, srv1733819850=41, srv1241283291=14, srv674177631=83, srv1979048046=55, srv236686949=66, srv802327564=93, srv1843855291=45, srv266779870=68, srv1949057542=51, srv1186183717=8, srv1314850117=19, srv1690735239=39, srv462289127=77, srv1929843505=50, srv277438289=70, srv1771913450=42, srv925895287=99, srv1728833013=40, srv2139930460=63, srv1330871135=22, srv730172652=85, srv891639221=96, srv122403027=10, srv123630834=12, srv2070018910=57, srv498613265=78, srv2042028071=56, srv816435379=94, srv1417777656=26, srv1964445756=53, srv1064898369=3, srv1255003997=17, srv1456258571=32, srv1328359830=20, srv796262840=92, srv735352436=87, srv761697121=89, srv222215344=65, srv2076113396=58, srv1807850272=43, srv773406401=90, srv660813612=82, srv1253879304=16, srv1188096529=9, srv118163806=7, srv1608408345=37, srv755664355=88, srv2133521317=61, srv1052543222=2, srv1120450366=6, srv1235885708=11, srv1281307152=18, srv1084027294=4, srv1446747859=31, srv1915227060=49, srv326102508=72, srv1887501821=48, srv1625345084=38, srv1813477927=44, srv41561453=75, srv266181425=67, srv785455784=91, srv358366243=73, srv363078329=74, srv1533418984=34, srv42824822=76, srv1045353445=1, srv1950770020=52, srv1439341820=29, srv1242911035=15, srv2080638778=59, srv1494859801=33, srv196673041=54, srv731901515=86, srv1350736448=23, srv1119491812=5, srv1861609016=46, srv209338870=60, srv218667123=64, srv1330557898=21, srv1438011461=28, srv499592528=79, srv1441587821=30, srv1427690977=27, srv311665603=71, srv1236977944=13, srv597020934=81, srv56071373=80, srv896951059=97, srv916422822=98, srv1548148263=35, srv701771231=84, srv1362087693=24, srv1878977787=47, srv1593548918=36, srv1027895821=0, srv276298898=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-05-03 02:01:12,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-05-03 02:01:12,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-05-03 02:01:12,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-05-03 02:01:12,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-05-03 02:01:12,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-05-03 02:01:12,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-05-03 02:01:12,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-05-03 02:01:12,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-05-03 02:01:12,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-05-03 02:01:12,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-05-03 02:01:12,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-05-03 02:01:12,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-05-03 02:01:12,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-05-03 02:01:12,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-05-03 02:01:12,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-05-03 02:01:12,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-05-03 02:01:12,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-05-03 02:01:12,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-05-03 02:01:12,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-05-03 02:01:12,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-05-03 02:01:12,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-05-03 02:01:12,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-05-03 02:01:12,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-05-03 02:01:12,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-05-03 02:01:12,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-05-03 02:01:12,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-05-03 02:01:12,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-05-03 02:01:12,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-05-03 02:01:12,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-05-03 02:01:12,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-05-03 02:01:12,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-05-03 02:01:12,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-05-03 02:01:12,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-05-03 02:01:12,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-05-03 02:01:12,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-05-03 02:01:12,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-05-03 02:01:12,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-05-03 02:01:12,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-05-03 02:01:12,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-05-03 02:01:12,931 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.979690117040589E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:01:12,931 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1052543222:83 , srv1120450366:83 , srv1328359830:83 , srv1441587821:83 , srv1446747859:83 , srv1533418984:83 , srv1949057542:83 , srv1950770020:83 , srv735352436:83 , srv755664355:83 , srv796262840:83 , srv876826478:83 , srv1045353445:84 , srv1064898369:84 , srv118163806:84 , srv1186183717:84 , srv122403027:84 , srv1235885708:84 , srv123630834:84 , srv1241283291:84 , srv1242911035:84 , srv1253879304:84 , srv1255003997:84 , srv1281307152:84 , srv1314850117:84 , srv1330557898:84 , srv1330871135:84 , srv1350736448:84 , srv1362087693:84 , srv1401279790:84 , srv1417777656:84 , srv1427690977:84 , srv1438011461:84 , srv1439341820:84 , srv1456258571:84 , srv1494859801:84 , srv1548148263:84 , srv1593548918:84 , srv1625345084:84 , srv1690735239:84 , srv1728833013:84 , srv1733819850:84 , srv1771913450:84 , srv1807850272:84 , srv1813477927:84 , srv1843855291:84 , srv1861609016:84 , srv1878977787:84 , srv1887501821:84 , srv1915227060:84 , srv1929843505:84 , srv196673041:84 , srv1979048046:84 , srv2042028071:84 , srv2070018910:84 , srv2076113396:84 , srv2080638778:84 , srv209338870:84 , srv2133521317:84 , srv2135153034:84 , srv2139930460:84 , srv218667123:84 , srv236686949:84 , srv266181425:84 , srv266779870:84 , srv276298898:84 , srv277438289:84 , srv311665603:84 , srv326102508:84 , srv363078329:84 , srv41561453:84 , srv42824822:84 , srv462289127:84 , srv498613265:84 , srv499592528:84 , srv56071373:84 , srv597020934:84 , srv660813612:84 , srv674177631:84 , srv701771231:84 , srv730172652:84 , srv731901515:84 , srv773406401:84 , srv785455784:84 , srv802327564:84 , srv816435379:84 , srv891639221:84 , srv916422822:84 , srv1027895821:85 , srv1084027294:85 , srv1119491812:85 , srv1188096529:85 , srv1236977944:85 , srv1608408345:85 , srv1964445756:85 , srv222215344:85 , srv358366243:85 , srv761697121:85 , srv925895287:85 , srv896951059:685 } 2024-05-03 02:01:12,955 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=245 (was 231) - SystemLoadAverage LEAK? -, ProcessCount=175 (was 175), AvailableMemoryMB=2148 (was 2588) 2024-05-03 02:01:12,964 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=245, ProcessCount=175, AvailableMemoryMB=2147 2024-05-03 02:01:12,965 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-05-03 02:01:12,965 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-05-03 02:01:12,976 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1035372481:84 , srv1047410023:84 , srv1084730913:84 , srv1095752844:84 , srv1099419314:84 , srv11045671:84 , srv1109796987:84 , srv1156684034:84 , srv1184183059:84 , srv1188345524:84 , srv1232999957:84 , srv1380069145:84 , srv1607870855:84 , srv1649803933:84 , srv1864600231:84 , srv1905491527:84 , srv1916595139:84 , srv2036839602:84 , srv2068212276:84 , srv2142425645:84 , srv228391266:84 , srv265284142:84 , srv292472172:84 , srv370397599:84 , srv407934998:84 , srv491746177:84 , srv867911444:84 , srv873155672:84 , srv923974754:84 , srv1398100675:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-05-03 02:01:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-05-03 02:01:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv292472172=23, srv1099419314=4, srv2036839602=18, srv491746177=26, srv2142425645=20, srv1380069145=11, srv873155672=28, srv1398100675=12, srv1084730913=2, srv2068212276=19, srv11045671=5, srv1905491527=16, srv1109796987=6, srv923974754=29, srv1156684034=7, srv228391266=21, srv265284142=22, srv1184183059=8, srv407934998=25, srv1035372481=0, srv867911444=27, srv370397599=24, srv1047410023=1, srv1916595139=17, srv1649803933=14, srv1607870855=13, srv1095752844=3, srv1864600231=15, srv1232999957=10, srv1188345524=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-05-03 02:01:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-05-03 02:01:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-05-03 02:01:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-05-03 02:01:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-05-03 02:01:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-05-03 02:01:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-05-03 02:01:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-05-03 02:01:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-05-03 02:01:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-05-03 02:01:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-05-03 02:01:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-05-03 02:01:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-05-03 02:01:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-05-03 02:01:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-05-03 02:01:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-05-03 02:01:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-05-03 02:01:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-05-03 02:01:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-05-03 02:01:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-05-03 02:01:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-05-03 02:01:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-05-03 02:01:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-05-03 02:01:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-05-03 02:01:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-05-03 02:01:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-05-03 02:01:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-05-03 02:01:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-05-03 02:01:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-05-03 02:01:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-05-03 02:01:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-05-03 02:01:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-05-03 02:01:13,011 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1376305550244308, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:01:13,011 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991262671264911, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1376305550244308, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-05-03 02:03:12,996 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9417191 different iterations. Found a solution that moves 1835 regions; Going from a computed imbalance of 0.991262671264911 to a new imbalance of 4.2829815420000605E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6796296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:03:13,001 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1035372481:90 , srv1047410023:90 , srv1084730913:90 , srv1095752844:90 , srv1099419314:90 , srv11045671:90 , srv1109796987:90 , srv1156684034:90 , srv1184183059:90 , srv1188345524:90 , srv1232999957:90 , srv1380069145:90 , srv1398100675:90 , srv1607870855:90 , srv1649803933:90 , srv1864600231:90 , srv1905491527:90 , srv1916595139:90 , srv2036839602:90 , srv2068212276:90 , srv2142425645:90 , srv228391266:90 , srv265284142:90 , srv292472172:90 , srv370397599:90 , srv407934998:90 , srv491746177:90 , srv867911444:90 , srv873155672:90 , srv923974754:90 } 2024-05-03 02:03:13,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-05-03 02:03:13,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv292472172=23, srv1099419314=4, srv2036839602=18, srv491746177=26, srv2142425645=20, srv1380069145=11, srv873155672=28, srv1398100675=12, srv1084730913=2, srv2068212276=19, srv11045671=5, srv1905491527=16, srv1109796987=6, srv923974754=29, srv1156684034=7, srv228391266=21, srv265284142=22, srv1184183059=8, srv407934998=25, srv1035372481=0, srv867911444=27, srv370397599=24, srv1047410023=1, srv1916595139=17, srv1649803933=14, srv1607870855=13, srv1095752844=3, srv1864600231=15, srv1232999957=10, srv1188345524=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-05-03 02:03:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-05-03 02:03:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-05-03 02:03:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-05-03 02:03:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-05-03 02:03:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-05-03 02:03:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-05-03 02:03:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-05-03 02:03:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-05-03 02:03:13,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-05-03 02:03:13,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-05-03 02:03:13,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-05-03 02:03:13,015 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-05-03 02:03:13,016 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1035372481:90 , srv1047410023:90 , srv1084730913:90 , srv1095752844:90 , srv1099419314:90 , srv11045671:90 , srv1109796987:90 , srv1156684034:90 , srv1184183059:90 , srv1188345524:90 , srv1232999957:90 , srv1380069145:90 , srv1398100675:90 , srv1607870855:90 , srv1649803933:90 , srv1864600231:90 , srv1905491527:90 , srv1916595139:90 , srv2036839602:90 , srv2068212276:90 , srv2142425645:90 , srv228391266:90 , srv265284142:90 , srv292472172:90 , srv370397599:90 , srv407934998:90 , srv491746177:90 , srv867911444:90 , srv873155672:90 , srv923974754:90 } 2024-05-03 02:03:13,028 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=175 (was 245), ProcessCount=174 (was 175), AvailableMemoryMB=1448 (was 2147)