2023-11-08 00:59:00,474 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-11-08 00:59:00,832 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-11-08 00:59:00,955 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-11-08 00:59:01,022 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-08 00:59:01,028 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-11-08 00:59:01,042 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=325, ProcessCount=166, AvailableMemoryMB=9091 2023-11-08 00:59:01,044 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-08 00:59:01,044 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-11-08 00:59:01,158 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv101273950:84 , srv1056620107:84 , srv1058741507:84 , srv1094234122:84 , srv1105276258:84 , srv1124601616:84 , srv1131900691:84 , srv1138278235:84 , srv1155149239:84 , srv1160929457:84 , srv1181363855:84 , srv118385361:84 , srv1195616663:84 , srv1209971013:84 , srv1246283365:84 , srv1249152359:84 , srv1293503551:84 , srv1294749177:84 , srv1355830008:84 , srv13903002:84 , srv1392308873:84 , srv1396583333:84 , srv1440454668:84 , srv1485333772:84 , srv1502722820:84 , srv1534906448:84 , srv1535190988:84 , srv1541545326:84 , srv1545199063:84 , srv1574759503:84 , srv158789502:84 , srv1600991510:84 , srv1612703247:84 , srv1614781338:84 , srv162463928:84 , srv1635247935:84 , srv1668692475:84 , srv1685611582:84 , srv1686719234:84 , srv1725530830:84 , srv172593255:84 , srv176059515:84 , srv1822676675:84 , srv1824779946:84 , srv182598673:84 , srv1837060434:84 , srv1837476195:84 , srv1906906968:84 , srv1959482051:84 , srv1973492066:84 , srv1990367699:84 , srv2011356909:84 , srv2012449684:84 , srv2015706071:84 , srv2028486534:84 , srv2092571675:84 , srv2095964604:84 , srv2103528444:84 , srv2141365112:84 , srv219658916:84 , srv222936732:84 , srv231522939:84 , srv237888471:84 , srv273189068:84 , srv307173054:84 , srv351715543:84 , srv352439635:84 , srv37624425:84 , srv401978960:84 , srv402525757:84 , srv441835152:84 , srv483873083:84 , srv500616086:84 , srv516840582:84 , srv526078705:84 , srv533771762:84 , srv535103144:84 , srv538970419:84 , srv540597585:84 , srv559960448:84 , srv57460584:84 , srv57676416:84 , srv586777050:84 , srv603529532:84 , srv607053068:84 , srv608290933:84 , srv630433252:84 , srv649777997:84 , srv688421040:84 , srv711488615:84 , srv72897159:84 , srv7755875:84 , srv77743010:84 , srv886469689:84 , srv916126027:84 , srv936752450:84 , srv943923696:84 , srv948071318:84 , srv966925705:84 , srv1137226268:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-11-08 00:59:01,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-08 00:59:01,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv172593255=41, srv1485333772=24, srv608290933=86, srv176059515=42, srv1094234122=3, srv2092571675=56, srv219658916=60, srv1294749177=18, srv231522939=62, srv182598673=45, srv526078705=75, srv1131900691=6, srv1105276258=4, srv1246283365=15, srv441835152=71, srv7755875=92, srv603529532=84, srv948071318=98, srv711488615=90, srv72897159=91, srv559960448=80, srv1906906968=48, srv1502722820=25, srv1541545326=28, srv1685611582=38, srv307173054=65, srv538970419=78, srv1837476195=47, srv966925705=99, srv1990367699=51, srv1058741507=2, srv162463928=35, srv688421040=89, srv1973492066=50, srv483873083=72, srv273189068=64, srv402525757=70, srv158789502=31, srv1195616663=13, srv1614781338=34, srv1440454668=23, srv1534906448=26, srv2141365112=59, srv352439635=67, srv500616086=73, srv630433252=87, srv1137226268=7, srv1160929457=10, srv1293503551=17, srv535103144=77, srv916126027=95, srv1725530830=40, srv2012449684=53, srv1600991510=32, srv57460584=81, srv2095964604=57, srv936752450=96, srv1545199063=29, srv1824779946=44, srv1249152359=16, srv1959482051=49, srv37624425=68, srv1392308873=21, srv533771762=76, srv1668692475=37, srv1837060434=46, srv1138278235=8, srv222936732=61, srv649777997=88, srv1612703247=33, srv1355830008=19, srv1686719234=39, srv237888471=63, srv13903002=20, srv1155149239=9, srv607053068=85, srv2011356909=52, srv351715543=66, srv586777050=83, srv943923696=97, srv1056620107=1, srv1181363855=11, srv2028486534=55, srv77743010=93, srv1574759503=30, srv1535190988=27, srv540597585=79, srv1822676675=43, srv401978960=69, srv886469689=94, srv2103528444=58, srv516840582=74, srv1635247935=36, srv57676416=82, srv1209971013=14, srv2015706071=54, srv101273950=0, srv1124601616=5, srv118385361=12, srv1396583333=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-08 00:59:01,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-08 00:59:01,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-08 00:59:01,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-08 00:59:01,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-08 00:59:01,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-08 00:59:01,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-08 00:59:01,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-08 00:59:01,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-08 00:59:01,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-08 00:59:01,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-08 00:59:01,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-08 00:59:01,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-08 00:59:01,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-08 00:59:01,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-08 00:59:01,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-08 00:59:01,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-08 00:59:01,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-08 00:59:01,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-08 00:59:01,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-08 00:59:01,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-08 00:59:01,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-08 00:59:01,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-08 00:59:01,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-08 00:59:01,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-08 00:59:01,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-08 00:59:01,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-08 00:59:01,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-08 00:59:01,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-08 00:59:01,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-08 00:59:01,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-08 00:59:01,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-08 00:59:01,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-08 00:59:01,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-08 00:59:01,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-08 00:59:01,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-08 00:59:01,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-08 00:59:01,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-08 00:59:01,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-08 00:59:01,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-08 00:59:01,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-08 00:59:01,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-08 00:59:01,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-08 00:59:01,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-08 00:59:01,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-08 00:59:01,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-08 00:59:01,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-08 00:59:01,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-08 00:59:01,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-08 00:59:01,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-08 00:59:01,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-08 00:59:01,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-08 00:59:01,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-08 00:59:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-08 00:59:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-08 00:59:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-08 00:59:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-08 00:59:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-08 00:59:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-08 00:59:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-08 00:59:01,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-08 00:59:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-08 00:59:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-08 00:59:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-08 00:59:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-08 00:59:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-08 00:59:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-08 00:59:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-08 00:59:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-08 00:59:01,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-08 00:59:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-08 00:59:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-08 00:59:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-08 00:59:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-08 00:59:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-08 00:59:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-08 00:59:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-08 00:59:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-08 00:59:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-08 00:59:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-08 00:59:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-08 00:59:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-08 00:59:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-08 00:59:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-08 00:59:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-08 00:59:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-08 00:59:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-08 00:59:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-08 00:59:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-08 00:59:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-08 00:59:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-08 00:59:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-08 00:59:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-08 00:59:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-08 00:59:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-08 00:59:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-08 00:59:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-08 00:59:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-08 00:59:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-08 00:59:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-08 00:59:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-08 00:59:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-08 00:59:01,471 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 00:59:01,472 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-08 00:59:01,472 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-08 00:59:01,809 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 503 ms to try 5000 different iterations. Found a solution that moves 4726 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17610742873133545. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040848953317747384, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05857645344221144, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5251111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040848953317747384, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17125, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23816666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 00:59:01,824 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv535103144:81 , srv101273950:82 , srv1209971013:82 , srv1355830008:82 , srv158789502:82 , srv2011356909:82 , srv2095964604:82 , srv943923696:82 , srv176059515:83 , srv307173054:83 , srv77743010:83 , srv1056620107:84 , srv1124601616:84 , srv1131900691:84 , srv1160929457:84 , srv1294749177:84 , srv1612703247:84 , srv162463928:84 , srv1668692475:84 , srv1824779946:84 , srv401978960:84 , srv57460584:84 , srv936752450:84 , srv1094234122:85 , srv1249152359:85 , srv1293503551:85 , srv1392308873:85 , srv1535190988:85 , srv1600991510:85 , srv1614781338:85 , srv1837060434:85 , srv1837476195:85 , srv1959482051:85 , srv1973492066:85 , srv2012449684:85 , srv2092571675:85 , srv2141365112:85 , srv352439635:85 , srv37624425:85 , srv559960448:85 , srv607053068:85 , srv630433252:85 , srv1058741507:86 , srv1155149239:86 , srv1195616663:86 , srv1396583333:86 , srv1534906448:86 , srv1545199063:86 , srv1686719234:86 , srv1725530830:86 , srv351715543:86 , srv483873083:86 , srv500616086:86 , srv586777050:86 , srv608290933:86 , srv72897159:86 , srv916126027:86 , srv1246283365:87 , srv1485333772:87 , srv1990367699:87 , srv2028486534:87 , srv219658916:87 , srv222936732:87 , srv231522939:87 , srv273189068:87 , srv441835152:87 , srv538970419:87 , srv57676416:87 , srv711488615:87 , srv948071318:87 , srv966925705:87 , srv1105276258:88 , srv118385361:88 , srv1502722820:88 , srv2103528444:88 , srv237888471:88 , srv402525757:88 , srv526078705:88 , srv533771762:88 , srv603529532:88 , srv1181363855:89 , srv1635247935:89 , srv1685611582:89 , srv182598673:89 , srv540597585:89 , srv649777997:89 , srv688421040:89 , srv1574759503:90 , srv172593255:90 , srv1906906968:90 , srv1138278235:91 , srv1440454668:91 , srv1822676675:91 , srv2015706071:91 , srv516840582:91 , srv7755875:92 , srv886469689:92 , srv1541545326:93 , srv13903002:95 , srv1137226268:453 } 2023-11-08 00:59:01,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-08 00:59:01,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv172593255=41, srv1485333772=24, srv608290933=86, srv176059515=42, srv1094234122=3, srv2092571675=56, srv219658916=60, srv1294749177=18, srv231522939=62, srv182598673=45, srv526078705=75, srv1131900691=6, srv1105276258=4, srv1246283365=15, srv441835152=71, srv7755875=92, srv603529532=84, srv948071318=98, srv711488615=90, srv72897159=91, srv559960448=80, srv1906906968=48, srv1502722820=25, srv1541545326=28, srv1685611582=38, srv307173054=65, srv538970419=78, srv1837476195=47, srv966925705=99, srv1990367699=51, srv1058741507=2, srv162463928=35, srv688421040=89, srv1973492066=50, srv483873083=72, srv273189068=64, srv402525757=70, srv158789502=31, srv1195616663=13, srv1614781338=34, srv1440454668=23, srv1534906448=26, srv2141365112=59, srv352439635=67, srv500616086=73, srv630433252=87, srv1137226268=7, srv1160929457=10, srv1293503551=17, srv535103144=77, srv916126027=95, srv1725530830=40, srv2012449684=53, srv1600991510=32, srv57460584=81, srv2095964604=57, srv936752450=96, srv1545199063=29, srv1824779946=44, srv1249152359=16, srv1959482051=49, srv37624425=68, srv1392308873=21, srv533771762=76, srv1668692475=37, srv1837060434=46, srv1138278235=8, srv222936732=61, srv649777997=88, srv1612703247=33, srv1355830008=19, srv1686719234=39, srv237888471=63, srv13903002=20, srv1155149239=9, srv607053068=85, srv2011356909=52, srv351715543=66, srv586777050=83, srv943923696=97, srv1056620107=1, srv1181363855=11, srv2028486534=55, srv77743010=93, srv1574759503=30, srv1535190988=27, srv540597585=79, srv1822676675=43, srv401978960=69, srv886469689=94, srv2103528444=58, srv516840582=74, srv1635247935=36, srv57676416=82, srv1209971013=14, srv2015706071=54, srv101273950=0, srv1124601616=5, srv118385361=12, srv1396583333=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-08 00:59:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-08 00:59:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-08 00:59:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-08 00:59:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-08 00:59:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-08 00:59:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-08 00:59:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-08 00:59:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-08 00:59:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-08 00:59:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-08 00:59:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-08 00:59:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-08 00:59:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-08 00:59:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-08 00:59:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-08 00:59:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-08 00:59:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-08 00:59:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-08 00:59:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-08 00:59:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-08 00:59:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-08 00:59:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-08 00:59:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-08 00:59:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-08 00:59:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-08 00:59:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-08 00:59:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-08 00:59:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-08 00:59:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-08 00:59:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-08 00:59:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-08 00:59:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-08 00:59:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-08 00:59:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-08 00:59:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-08 00:59:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-08 00:59:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-08 00:59:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-08 00:59:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-08 00:59:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-08 00:59:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-08 00:59:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-08 00:59:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-08 00:59:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-08 00:59:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-08 00:59:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-08 00:59:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-08 00:59:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-08 00:59:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-08 00:59:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-08 00:59:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-08 00:59:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-08 00:59:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-08 00:59:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-08 00:59:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-08 00:59:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-08 00:59:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-08 00:59:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-08 00:59:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-08 00:59:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-08 00:59:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-08 00:59:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-08 00:59:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-08 00:59:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-08 00:59:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-08 00:59:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-08 00:59:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-08 00:59:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-08 00:59:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-08 00:59:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-08 00:59:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-08 00:59:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-08 00:59:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-08 00:59:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-08 00:59:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-08 00:59:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-08 00:59:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-08 00:59:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-08 00:59:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-08 00:59:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-08 00:59:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-08 00:59:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-08 00:59:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-08 00:59:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-08 00:59:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-08 00:59:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-08 00:59:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-08 00:59:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-08 00:59:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-08 00:59:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-08 00:59:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-08 00:59:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-08 00:59:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-08 00:59:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-08 00:59:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-08 00:59:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-08 00:59:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-08 00:59:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-08 00:59:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-08 00:59:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-08 00:59:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-08 00:59:02,007 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040848953317747384, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05857645344221144, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040848953317747384, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17125, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23816666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 00:59:02,007 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-08 00:59:02,008 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17607433657204255, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040848953317747384, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05857645344221144, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040848953317747384, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17125, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23816666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-08 00:59:02,191 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 303 ms to try 5000 different iterations. Found a solution that moves 2607 regions; Going from a computed imbalance of 0.17607433657204255 to a new imbalance of 0.013994769758506168. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02441279452920696); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.042150390784927605, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2896666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02441279452920696); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042666666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 00:59:02,200 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv13903002:77 , srv1541545326:78 , srv886469689:78 , srv1440454668:79 , srv2015706071:80 , srv1138278235:81 , srv1181363855:81 , srv1574759503:81 , srv1822676675:81 , srv540597585:81 , srv1105276258:83 , srv182598673:83 , srv1906906968:83 , srv608290933:83 , srv649777997:83 , srv7755875:83 , srv948071318:83 , srv1195616663:84 , srv1685611582:84 , srv1990367699:84 , srv2103528444:84 , srv222936732:84 , srv441835152:84 , srv483873083:84 , srv526078705:84 , srv533771762:84 , srv603529532:84 , srv118385361:85 , srv1246283365:85 , srv1293503551:85 , srv1545199063:85 , srv1600991510:85 , srv1614781338:85 , srv1635247935:85 , srv1837476195:85 , srv1959482051:85 , srv231522939:85 , srv402525757:85 , srv516840582:85 , srv630433252:85 , srv688421040:85 , srv916126027:85 , srv1056620107:86 , srv1094234122:86 , srv1124601616:86 , srv1155149239:86 , srv1294749177:86 , srv1396583333:86 , srv1485333772:86 , srv1534906448:86 , srv1668692475:86 , srv1725530830:86 , srv172593255:86 , srv1837060434:86 , srv1973492066:86 , srv2012449684:86 , srv2028486534:86 , srv2092571675:86 , srv237888471:86 , srv352439635:86 , srv401978960:86 , srv538970419:86 , srv559960448:86 , srv57460584:86 , srv586777050:86 , srv607053068:86 , srv72897159:86 , srv966925705:86 , srv1058741507:87 , srv1160929457:87 , srv1249152359:87 , srv1392308873:87 , srv1502722820:87 , srv1535190988:87 , srv176059515:87 , srv1824779946:87 , srv2141365112:87 , srv219658916:87 , srv273189068:87 , srv351715543:87 , srv57676416:87 , srv77743010:87 , srv101273950:88 , srv158789502:88 , srv1612703247:88 , srv162463928:88 , srv1686719234:88 , srv307173054:88 , srv500616086:88 , srv711488615:88 , srv1131900691:89 , srv37624425:89 , srv936752450:89 , srv1355830008:90 , srv2011356909:90 , srv943923696:90 , srv1209971013:91 , srv2095964604:91 , srv535103144:92 , srv1137226268:538 } 2023-11-08 00:59:02,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-08 00:59:02,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv172593255=41, srv1485333772=24, srv608290933=86, srv176059515=42, srv1094234122=3, srv2092571675=56, srv219658916=60, srv1294749177=18, srv231522939=62, srv182598673=45, srv526078705=75, srv1131900691=6, srv1105276258=4, srv1246283365=15, srv441835152=71, srv7755875=92, srv603529532=84, srv948071318=98, srv711488615=90, srv72897159=91, srv559960448=80, srv1906906968=48, srv1502722820=25, srv1541545326=28, srv1685611582=38, srv307173054=65, srv538970419=78, srv1837476195=47, srv966925705=99, srv1990367699=51, srv1058741507=2, srv162463928=35, srv688421040=89, srv1973492066=50, srv483873083=72, srv273189068=64, srv402525757=70, srv158789502=31, srv1195616663=13, srv1614781338=34, srv1440454668=23, srv1534906448=26, srv2141365112=59, srv352439635=67, srv500616086=73, srv630433252=87, srv1137226268=7, srv1160929457=10, srv1293503551=17, srv535103144=77, srv916126027=95, srv1725530830=40, srv2012449684=53, srv1600991510=32, srv57460584=81, srv2095964604=57, srv936752450=96, srv1545199063=29, srv1824779946=44, srv1249152359=16, srv1959482051=49, srv37624425=68, srv1392308873=21, srv533771762=76, srv1668692475=37, srv1837060434=46, srv1138278235=8, srv222936732=61, srv649777997=88, srv1612703247=33, srv1355830008=19, srv1686719234=39, srv237888471=63, srv13903002=20, srv1155149239=9, srv607053068=85, srv2011356909=52, srv351715543=66, srv586777050=83, srv943923696=97, srv1056620107=1, srv1181363855=11, srv2028486534=55, srv77743010=93, srv1574759503=30, srv1535190988=27, srv540597585=79, srv1822676675=43, srv401978960=69, srv886469689=94, srv2103528444=58, srv516840582=74, srv1635247935=36, srv57676416=82, srv1209971013=14, srv2015706071=54, srv101273950=0, srv1124601616=5, srv118385361=12, srv1396583333=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-08 00:59:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-08 00:59:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-08 00:59:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-08 00:59:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-08 00:59:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-08 00:59:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-08 00:59:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-08 00:59:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-08 00:59:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-08 00:59:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-08 00:59:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-08 00:59:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-08 00:59:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-08 00:59:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-08 00:59:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-08 00:59:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-08 00:59:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-08 00:59:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-08 00:59:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-08 00:59:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-08 00:59:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-08 00:59:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-08 00:59:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-08 00:59:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-08 00:59:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-08 00:59:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-08 00:59:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-08 00:59:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-08 00:59:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-08 00:59:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-08 00:59:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-08 00:59:02,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-08 00:59:02,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-08 00:59:02,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-08 00:59:02,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-08 00:59:02,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-08 00:59:02,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-08 00:59:02,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-08 00:59:02,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-08 00:59:02,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-08 00:59:02,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-08 00:59:02,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-08 00:59:02,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-08 00:59:02,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-08 00:59:02,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-08 00:59:02,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-08 00:59:02,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-08 00:59:02,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-08 00:59:02,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-08 00:59:02,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-08 00:59:02,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-08 00:59:02,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-08 00:59:02,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-08 00:59:02,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-08 00:59:02,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-08 00:59:02,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-08 00:59:02,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-08 00:59:02,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-08 00:59:02,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-08 00:59:02,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-08 00:59:02,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-08 00:59:02,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-08 00:59:02,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-08 00:59:02,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-08 00:59:02,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-08 00:59:02,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-08 00:59:02,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-08 00:59:02,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-08 00:59:02,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-08 00:59:02,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-08 00:59:02,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-08 00:59:02,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-08 00:59:02,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-08 00:59:02,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-08 00:59:02,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-08 00:59:02,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-08 00:59:02,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-08 00:59:02,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-08 00:59:02,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-08 00:59:02,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-08 00:59:02,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-08 00:59:02,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-08 00:59:02,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-08 00:59:02,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-08 00:59:02,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-08 00:59:02,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-08 00:59:02,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-08 00:59:02,335 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02441279452920696); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.042150390784927605, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02441279452920696); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042666666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 00:59:02,336 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-08 00:59:02,336 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013976515154342688, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02441279452920696); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.042150390784927605, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02441279452920696); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042666666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-08 00:59:02,505 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 262 ms to try 5000 different iterations. Found a solution that moves 1027 regions; Going from a computed imbalance of 0.013976515154342688 to a new imbalance of 6.979726776697075E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006905558918946408); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009404251969541318); MoveCostFunction : (multiplier=7.0, imbalance=0.11411111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006905558918946408); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 00:59:02,506 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv516840582:81 , srv1686719234:83 , srv172593255:83 , srv237888471:83 , srv57676416:83 , srv711488615:83 , srv7755875:83 , srv1058741507:84 , srv118385361:84 , srv1502722820:84 , srv1685611582:84 , srv1906906968:84 , srv2028486534:84 , srv2095964604:84 , srv219658916:84 , srv273189068:84 , srv351715543:84 , srv37624425:84 , srv402525757:84 , srv500616086:84 , srv538970419:84 , srv603529532:84 , srv688421040:84 , srv966925705:84 , srv1094234122:85 , srv1131900691:85 , srv1138278235:85 , srv1209971013:85 , srv1246283365:85 , srv1249152359:85 , srv1355830008:85 , srv13903002:85 , srv1392308873:85 , srv1485333772:85 , srv1535190988:85 , srv1545199063:85 , srv1574759503:85 , srv1614781338:85 , srv162463928:85 , srv1725530830:85 , srv1824779946:85 , srv2141365112:85 , srv222936732:85 , srv526078705:85 , srv533771762:85 , srv535103144:85 , srv586777050:85 , srv649777997:85 , srv72897159:85 , srv1105276258:86 , srv1155149239:86 , srv1160929457:86 , srv1396583333:86 , srv1534906448:86 , srv1541545326:86 , srv1612703247:86 , srv1822676675:86 , srv182598673:86 , srv1837060434:86 , srv1959482051:86 , srv1990367699:86 , srv2011356909:86 , srv2015706071:86 , srv2103528444:86 , srv231522939:86 , srv352439635:86 , srv441835152:86 , srv886469689:86 , srv916126027:86 , srv936752450:86 , srv943923696:86 , srv1056620107:87 , srv1294749177:87 , srv1440454668:87 , srv158789502:87 , srv1600991510:87 , srv1635247935:87 , srv176059515:87 , srv1837476195:87 , srv1973492066:87 , srv2012449684:87 , srv2092571675:87 , srv307173054:87 , srv483873083:87 , srv540597585:87 , srv559960448:87 , srv607053068:87 , srv630433252:87 , srv948071318:87 , srv101273950:88 , srv1124601616:88 , srv1181363855:88 , srv1195616663:88 , srv1668692475:88 , srv401978960:88 , srv57460584:88 , srv77743010:88 , srv1293503551:89 , srv608290933:89 , srv1137226268:528 } 2023-11-08 00:59:02,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-08 00:59:02,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv172593255=41, srv1485333772=24, srv608290933=86, srv176059515=42, srv1094234122=3, srv2092571675=56, srv219658916=60, srv1294749177=18, srv231522939=62, srv182598673=45, srv526078705=75, srv1131900691=6, srv1105276258=4, srv1246283365=15, srv441835152=71, srv7755875=92, srv603529532=84, srv948071318=98, srv711488615=90, srv72897159=91, srv559960448=80, srv1906906968=48, srv1502722820=25, srv1541545326=28, srv1685611582=38, srv307173054=65, srv538970419=78, srv1837476195=47, srv966925705=99, srv1990367699=51, srv1058741507=2, srv162463928=35, srv688421040=89, srv1973492066=50, srv483873083=72, srv273189068=64, srv402525757=70, srv158789502=31, srv1195616663=13, srv1614781338=34, srv1440454668=23, srv1534906448=26, srv2141365112=59, srv352439635=67, srv500616086=73, srv630433252=87, srv1137226268=7, srv1160929457=10, srv1293503551=17, srv535103144=77, srv916126027=95, srv1725530830=40, srv2012449684=53, srv1600991510=32, srv57460584=81, srv2095964604=57, srv936752450=96, srv1545199063=29, srv1824779946=44, srv1249152359=16, srv1959482051=49, srv37624425=68, srv1392308873=21, srv533771762=76, srv1668692475=37, srv1837060434=46, srv1138278235=8, srv222936732=61, srv649777997=88, srv1612703247=33, srv1355830008=19, srv1686719234=39, srv237888471=63, srv13903002=20, srv1155149239=9, srv607053068=85, srv2011356909=52, srv351715543=66, srv586777050=83, srv943923696=97, srv1056620107=1, srv1181363855=11, srv2028486534=55, srv77743010=93, srv1574759503=30, srv1535190988=27, srv540597585=79, srv1822676675=43, srv401978960=69, srv886469689=94, srv2103528444=58, srv516840582=74, srv1635247935=36, srv57676416=82, srv1209971013=14, srv2015706071=54, srv101273950=0, srv1124601616=5, srv118385361=12, srv1396583333=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-08 00:59:02,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-08 00:59:02,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-08 00:59:02,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-08 00:59:02,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-08 00:59:02,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-08 00:59:02,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-08 00:59:02,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-08 00:59:02,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-08 00:59:02,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-08 00:59:02,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-08 00:59:02,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-08 00:59:02,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-08 00:59:02,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-08 00:59:02,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-08 00:59:02,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-08 00:59:02,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-08 00:59:02,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-08 00:59:02,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-08 00:59:02,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-08 00:59:02,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-08 00:59:02,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-08 00:59:02,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-08 00:59:02,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-08 00:59:02,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-08 00:59:02,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-08 00:59:02,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-08 00:59:02,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-08 00:59:02,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-08 00:59:02,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-08 00:59:02,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-08 00:59:02,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-08 00:59:02,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-08 00:59:02,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-08 00:59:02,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-08 00:59:02,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-08 00:59:02,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-08 00:59:02,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-08 00:59:02,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-08 00:59:02,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-08 00:59:02,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-08 00:59:02,619 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006905558918946408); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009404251969541318); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006905558918946408); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 00:59:02,619 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-08 00:59:02,620 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.907814699689433E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006905558918946408); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009404251969541318); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006905558918946408); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-08 00:59:02,868 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 331 ms to try 5000 different iterations. Found a solution that moves 324 regions; Going from a computed imbalance of 6.907814699689433E-4 to a new imbalance of 2.9783138251359097E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.036, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 00:59:02,869 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1635247935:81 , srv1293503551:83 , srv1502722820:83 , srv1534906448:83 , srv936752450:83 , srv101273950:84 , srv1124601616:84 , srv1131900691:84 , srv1155149239:84 , srv1181363855:84 , srv1209971013:84 , srv13903002:84 , srv1396583333:84 , srv1485333772:84 , srv1535190988:84 , srv1612703247:84 , srv1668692475:84 , srv1686719234:84 , srv172593255:84 , srv1822676675:84 , srv182598673:84 , srv1973492066:84 , srv1990367699:84 , srv2011356909:84 , srv2012449684:84 , srv2092571675:84 , srv2103528444:84 , srv219658916:84 , srv237888471:84 , srv273189068:84 , srv307173054:84 , srv352439635:84 , srv37624425:84 , srv401978960:84 , srv441835152:84 , srv500616086:84 , srv535103144:84 , srv559960448:84 , srv57460584:84 , srv586777050:84 , srv607053068:84 , srv608290933:84 , srv688421040:84 , srv711488615:84 , srv7755875:84 , srv77743010:84 , srv943923696:84 , srv1056620107:85 , srv1058741507:85 , srv1105276258:85 , srv1138278235:85 , srv1160929457:85 , srv1195616663:85 , srv1246283365:85 , srv1249152359:85 , srv1294749177:85 , srv1355830008:85 , srv1440454668:85 , srv158789502:85 , srv162463928:85 , srv1725530830:85 , srv176059515:85 , srv1837060434:85 , srv1837476195:85 , srv1906906968:85 , srv2015706071:85 , srv2028486534:85 , srv2095964604:85 , srv2141365112:85 , srv231522939:85 , srv351715543:85 , srv402525757:85 , srv483873083:85 , srv516840582:85 , srv526078705:85 , srv533771762:85 , srv538970419:85 , srv540597585:85 , srv57676416:85 , srv630433252:85 , srv649777997:85 , srv72897159:85 , srv916126027:85 , srv948071318:85 , srv966925705:85 , srv1094234122:86 , srv118385361:86 , srv1392308873:86 , srv1541545326:86 , srv1545199063:86 , srv1600991510:86 , srv1685611582:86 , srv1824779946:86 , srv1959482051:86 , srv222936732:86 , srv603529532:86 , srv886469689:86 , srv1574759503:87 , srv1614781338:87 , srv1137226268:623 } 2023-11-08 00:59:02,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-08 00:59:02,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv172593255=41, srv1485333772=24, srv608290933=86, srv176059515=42, srv1094234122=3, srv2092571675=56, srv219658916=60, srv1294749177=18, srv231522939=62, srv182598673=45, srv526078705=75, srv1131900691=6, srv1105276258=4, srv1246283365=15, srv441835152=71, srv7755875=92, srv603529532=84, srv948071318=98, srv711488615=90, srv72897159=91, srv559960448=80, srv1906906968=48, srv1502722820=25, srv1541545326=28, srv1685611582=38, srv307173054=65, srv538970419=78, srv1837476195=47, srv966925705=99, srv1990367699=51, srv1058741507=2, srv162463928=35, srv688421040=89, srv1973492066=50, srv483873083=72, srv273189068=64, srv402525757=70, srv158789502=31, srv1195616663=13, srv1614781338=34, srv1440454668=23, srv1534906448=26, srv2141365112=59, srv352439635=67, srv500616086=73, srv630433252=87, srv1137226268=7, srv1160929457=10, srv1293503551=17, srv535103144=77, srv916126027=95, srv1725530830=40, srv2012449684=53, srv1600991510=32, srv57460584=81, srv2095964604=57, srv936752450=96, srv1545199063=29, srv1824779946=44, srv1249152359=16, srv1959482051=49, srv37624425=68, srv1392308873=21, srv533771762=76, srv1668692475=37, srv1837060434=46, srv1138278235=8, srv222936732=61, srv649777997=88, srv1612703247=33, srv1355830008=19, srv1686719234=39, srv237888471=63, srv13903002=20, srv1155149239=9, srv607053068=85, srv2011356909=52, srv351715543=66, srv586777050=83, srv943923696=97, srv1056620107=1, srv1181363855=11, srv2028486534=55, srv77743010=93, srv1574759503=30, srv1535190988=27, srv540597585=79, srv1822676675=43, srv401978960=69, srv886469689=94, srv2103528444=58, srv516840582=74, srv1635247935=36, srv57676416=82, srv1209971013=14, srv2015706071=54, srv101273950=0, srv1124601616=5, srv118385361=12, srv1396583333=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-08 00:59:02,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-08 00:59:02,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-08 00:59:02,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-08 00:59:02,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-08 00:59:02,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-08 00:59:02,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-08 00:59:02,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-08 00:59:02,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-08 00:59:02,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-08 00:59:02,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-08 00:59:02,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-08 00:59:02,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-08 00:59:02,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-08 00:59:02,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-08 00:59:02,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-08 00:59:02,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-08 00:59:02,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-08 00:59:02,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-08 00:59:02,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-08 00:59:02,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-08 00:59:02,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-08 00:59:02,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-08 00:59:02,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-08 00:59:02,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-08 00:59:02,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-08 00:59:02,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-08 00:59:02,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-08 00:59:02,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-08 00:59:02,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-08 00:59:02,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-08 00:59:02,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-08 00:59:02,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-08 00:59:02,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-08 00:59:02,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-08 00:59:02,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-08 00:59:02,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-08 00:59:02,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-08 00:59:02,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-08 00:59:02,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-08 00:59:02,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-08 00:59:02,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-08 00:59:02,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-08 00:59:02,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-08 00:59:02,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-08 00:59:02,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-08 00:59:02,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-08 00:59:02,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-08 00:59:02,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-08 00:59:02,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-08 00:59:02,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-08 00:59:02,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-08 00:59:02,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-08 00:59:02,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-08 00:59:02,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-08 00:59:02,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-08 00:59:02,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-08 00:59:02,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-08 00:59:02,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-08 00:59:02,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-08 00:59:02,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-08 00:59:02,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-08 00:59:02,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-08 00:59:02,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-08 00:59:02,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-08 00:59:02,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-08 00:59:02,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-08 00:59:02,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-08 00:59:02,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-08 00:59:02,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-08 00:59:02,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-08 00:59:02,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-08 00:59:02,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-08 00:59:02,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-08 00:59:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-08 00:59:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-08 00:59:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-08 00:59:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-08 00:59:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-08 00:59:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-08 00:59:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-08 00:59:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-08 00:59:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-08 00:59:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-08 00:59:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-08 00:59:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-08 00:59:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-08 00:59:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-08 00:59:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-08 00:59:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-08 00:59:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-08 00:59:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-08 00:59:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-08 00:59:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-08 00:59:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-08 00:59:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-08 00:59:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-08 00:59:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-08 00:59:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-08 00:59:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-08 00:59:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-08 00:59:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-08 00:59:02,984 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 00:59:02,984 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-08 00:59:02,985 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.7514441761536726E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-08 00:59:03,175 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 268 ms to try 5000 different iterations. Found a solution that moves 31 regions; Going from a computed imbalance of 2.7514441761536726E-5 to a new imbalance of 7.604824139770872E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0034444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 00:59:03,175 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv118385361:83 , srv1195616663:83 , srv1392308873:83 , srv1541545326:83 , srv1574759503:83 , srv1600991510:83 , srv1685611582:83 , srv172593255:83 , srv231522939:83 , srv101273950:84 , srv1056620107:84 , srv1058741507:84 , srv1094234122:84 , srv1105276258:84 , srv1124601616:84 , srv1131900691:84 , srv1138278235:84 , srv1155149239:84 , srv1160929457:84 , srv1181363855:84 , srv1209971013:84 , srv1246283365:84 , srv1249152359:84 , srv1293503551:84 , srv1294749177:84 , srv1355830008:84 , srv1396583333:84 , srv1440454668:84 , srv1485333772:84 , srv1502722820:84 , srv1545199063:84 , srv158789502:84 , srv1612703247:84 , srv1614781338:84 , srv162463928:84 , srv1635247935:84 , srv1668692475:84 , srv1725530830:84 , srv176059515:84 , srv1822676675:84 , srv1824779946:84 , srv182598673:84 , srv1837060434:84 , srv1837476195:84 , srv1906906968:84 , srv1959482051:84 , srv1973492066:84 , srv2011356909:84 , srv2012449684:84 , srv2015706071:84 , srv2028486534:84 , srv2092571675:84 , srv2095964604:84 , srv2103528444:84 , srv2141365112:84 , srv219658916:84 , srv222936732:84 , srv273189068:84 , srv307173054:84 , srv351715543:84 , srv37624425:84 , srv401978960:84 , srv402525757:84 , srv483873083:84 , srv500616086:84 , srv516840582:84 , srv526078705:84 , srv533771762:84 , srv535103144:84 , srv538970419:84 , srv540597585:84 , srv559960448:84 , srv57460584:84 , srv57676416:84 , srv603529532:84 , srv607053068:84 , srv608290933:84 , srv630433252:84 , srv649777997:84 , srv688421040:84 , srv711488615:84 , srv72897159:84 , srv7755875:84 , srv77743010:84 , srv886469689:84 , srv916126027:84 , srv936752450:84 , srv943923696:84 , srv948071318:84 , srv966925705:84 , srv13903002:85 , srv1534906448:85 , srv1535190988:85 , srv1686719234:85 , srv1990367699:85 , srv237888471:85 , srv352439635:85 , srv441835152:85 , srv586777050:85 , srv1137226268:684 } 2023-11-08 00:59:03,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-08 00:59:03,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv172593255=41, srv1485333772=24, srv608290933=86, srv176059515=42, srv1094234122=3, srv2092571675=56, srv219658916=60, srv1294749177=18, srv231522939=62, srv182598673=45, srv526078705=75, srv1131900691=6, srv1105276258=4, srv1246283365=15, srv441835152=71, srv7755875=92, srv603529532=84, srv948071318=98, srv711488615=90, srv72897159=91, srv559960448=80, srv1906906968=48, srv1502722820=25, srv1541545326=28, srv1685611582=38, srv307173054=65, srv538970419=78, srv1837476195=47, srv966925705=99, srv1990367699=51, srv1058741507=2, srv162463928=35, srv688421040=89, srv1973492066=50, srv483873083=72, srv273189068=64, srv402525757=70, srv158789502=31, srv1195616663=13, srv1614781338=34, srv1440454668=23, srv1534906448=26, srv2141365112=59, srv352439635=67, srv500616086=73, srv630433252=87, srv1137226268=7, srv1160929457=10, srv1293503551=17, srv535103144=77, srv916126027=95, srv1725530830=40, srv2012449684=53, srv1600991510=32, srv57460584=81, srv2095964604=57, srv936752450=96, srv1545199063=29, srv1824779946=44, srv1249152359=16, srv1959482051=49, srv37624425=68, srv1392308873=21, srv533771762=76, srv1668692475=37, srv1837060434=46, srv1138278235=8, srv222936732=61, srv649777997=88, srv1612703247=33, srv1355830008=19, srv1686719234=39, srv237888471=63, srv13903002=20, srv1155149239=9, srv607053068=85, srv2011356909=52, srv351715543=66, srv586777050=83, srv943923696=97, srv1056620107=1, srv1181363855=11, srv2028486534=55, srv77743010=93, srv1574759503=30, srv1535190988=27, srv540597585=79, srv1822676675=43, srv401978960=69, srv886469689=94, srv2103528444=58, srv516840582=74, srv1635247935=36, srv57676416=82, srv1209971013=14, srv2015706071=54, srv101273950=0, srv1124601616=5, srv118385361=12, srv1396583333=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-08 00:59:03,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-08 00:59:03,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-08 00:59:03,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-08 00:59:03,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-08 00:59:03,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-08 00:59:03,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-08 00:59:03,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-08 00:59:03,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-08 00:59:03,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-08 00:59:03,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-08 00:59:03,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-08 00:59:03,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-08 00:59:03,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-08 00:59:03,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-08 00:59:03,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-08 00:59:03,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-08 00:59:03,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-08 00:59:03,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-08 00:59:03,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-08 00:59:03,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-08 00:59:03,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-08 00:59:03,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-08 00:59:03,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-08 00:59:03,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-08 00:59:03,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-08 00:59:03,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-08 00:59:03,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-08 00:59:03,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-08 00:59:03,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-08 00:59:03,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-08 00:59:03,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-08 00:59:03,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-08 00:59:03,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-08 00:59:03,288 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 00:59:03,288 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv118385361:83 , srv1195616663:83 , srv1392308873:83 , srv1541545326:83 , srv1574759503:83 , srv1600991510:83 , srv1685611582:83 , srv172593255:83 , srv231522939:83 , srv101273950:84 , srv1056620107:84 , srv1058741507:84 , srv1094234122:84 , srv1105276258:84 , srv1124601616:84 , srv1131900691:84 , srv1138278235:84 , srv1155149239:84 , srv1160929457:84 , srv1181363855:84 , srv1209971013:84 , srv1246283365:84 , srv1249152359:84 , srv1293503551:84 , srv1294749177:84 , srv1355830008:84 , srv1396583333:84 , srv1440454668:84 , srv1485333772:84 , srv1502722820:84 , srv1545199063:84 , srv158789502:84 , srv1612703247:84 , srv1614781338:84 , srv162463928:84 , srv1635247935:84 , srv1668692475:84 , srv1725530830:84 , srv176059515:84 , srv1822676675:84 , srv1824779946:84 , srv182598673:84 , srv1837060434:84 , srv1837476195:84 , srv1906906968:84 , srv1959482051:84 , srv1973492066:84 , srv2011356909:84 , srv2012449684:84 , srv2015706071:84 , srv2028486534:84 , srv2092571675:84 , srv2095964604:84 , srv2103528444:84 , srv2141365112:84 , srv219658916:84 , srv222936732:84 , srv273189068:84 , srv307173054:84 , srv351715543:84 , srv37624425:84 , srv401978960:84 , srv402525757:84 , srv483873083:84 , srv500616086:84 , srv516840582:84 , srv526078705:84 , srv533771762:84 , srv535103144:84 , srv538970419:84 , srv540597585:84 , srv559960448:84 , srv57460584:84 , srv57676416:84 , srv603529532:84 , srv607053068:84 , srv608290933:84 , srv630433252:84 , srv649777997:84 , srv688421040:84 , srv711488615:84 , srv72897159:84 , srv7755875:84 , srv77743010:84 , srv886469689:84 , srv916126027:84 , srv936752450:84 , srv943923696:84 , srv948071318:84 , srv966925705:84 , srv13903002:85 , srv1534906448:85 , srv1535190988:85 , srv1686719234:85 , srv1990367699:85 , srv237888471:85 , srv352439635:85 , srv441835152:85 , srv586777050:85 , srv1137226268:684 } 2023-11-08 00:59:03,309 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=363 (was 325) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=8482 (was 9091) 2023-11-08 00:59:03,317 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=363, ProcessCount=166, AvailableMemoryMB=8482 2023-11-08 00:59:03,317 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-08 00:59:03,318 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-11-08 00:59:03,328 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100015403:84 , srv1070127928:84 , srv1115436184:84 , srv114962843:84 , srv115440593:84 , srv1348955171:84 , srv136826271:84 , srv1411648105:84 , srv1546916132:84 , srv1574192660:84 , srv1859938140:84 , srv1898980281:84 , srv191739728:84 , srv2049693332:84 , srv2077952792:84 , srv2094283480:84 , srv217763326:84 , srv24167767:84 , srv494096260:84 , srv49587129:84 , srv526870225:84 , srv579950166:84 , srv597146322:84 , srv612797207:84 , srv769637174:84 , srv801323638:84 , srv856118910:84 , srv910458570:84 , srv935343534:84 , srv457546967:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-11-08 00:59:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-08 00:59:03,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1115436184=2, srv457546967=18, srv526870225=21, srv1574192660=9, srv494096260=19, srv597146322=23, srv2049693332=13, srv579950166=22, srv1898980281=11, srv100015403=0, srv856118910=27, srv612797207=24, srv769637174=25, srv114962843=3, srv24167767=17, srv935343534=29, srv115440593=4, srv910458570=28, srv2094283480=15, srv1348955171=5, srv49587129=20, srv217763326=16, srv136826271=6, srv191739728=12, srv2077952792=14, srv1859938140=10, srv1411648105=7, srv801323638=26, srv1070127928=1, srv1546916132=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-08 00:59:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-08 00:59:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-08 00:59:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-08 00:59:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-08 00:59:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-08 00:59:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-08 00:59:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-08 00:59:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-08 00:59:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-08 00:59:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-08 00:59:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-11-08 00:59:03,352 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1526164973121682, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 00:59:03,352 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912673932863922, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1526164973121682, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-11-08 01:01:03,342 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9341546 different iterations. Found a solution that moves 1836 regions; Going from a computed imbalance of 0.9912673932863922 to a new imbalance of 4.2853155918867096E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.68, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 01:01:03,348 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv100015403:90 , srv1070127928:90 , srv1115436184:90 , srv114962843:90 , srv115440593:90 , srv1348955171:90 , srv136826271:90 , srv1411648105:90 , srv1546916132:90 , srv1574192660:90 , srv1859938140:90 , srv1898980281:90 , srv191739728:90 , srv2049693332:90 , srv2077952792:90 , srv2094283480:90 , srv217763326:90 , srv24167767:90 , srv457546967:90 , srv494096260:90 , srv49587129:90 , srv526870225:90 , srv579950166:90 , srv597146322:90 , srv612797207:90 , srv769637174:90 , srv801323638:90 , srv856118910:90 , srv910458570:90 , srv935343534:90 } 2023-11-08 01:01:03,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-08 01:01:03,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1115436184=2, srv457546967=18, srv526870225=21, srv1574192660=9, srv494096260=19, srv597146322=23, srv2049693332=13, srv579950166=22, srv1898980281=11, srv100015403=0, srv856118910=27, srv612797207=24, srv769637174=25, srv114962843=3, srv24167767=17, srv935343534=29, srv115440593=4, srv910458570=28, srv2094283480=15, srv1348955171=5, srv49587129=20, srv217763326=16, srv136826271=6, srv191739728=12, srv2077952792=14, srv1859938140=10, srv1411648105=7, srv801323638=26, srv1070127928=1, srv1546916132=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-08 01:01:03,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-08 01:01:03,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-08 01:01:03,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-08 01:01:03,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-08 01:01:03,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-08 01:01:03,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-08 01:01:03,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-08 01:01:03,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-08 01:01:03,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-08 01:01:03,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-08 01:01:03,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-08 01:01:03,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-11-08 01:01:03,366 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-08 01:01:03,366 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv100015403:90 , srv1070127928:90 , srv1115436184:90 , srv114962843:90 , srv115440593:90 , srv1348955171:90 , srv136826271:90 , srv1411648105:90 , srv1546916132:90 , srv1574192660:90 , srv1859938140:90 , srv1898980281:90 , srv191739728:90 , srv2049693332:90 , srv2077952792:90 , srv2094283480:90 , srv217763326:90 , srv24167767:90 , srv457546967:90 , srv494096260:90 , srv49587129:90 , srv526870225:90 , srv579950166:90 , srv597146322:90 , srv612797207:90 , srv769637174:90 , srv801323638:90 , srv856118910:90 , srv910458570:90 , srv935343534:90 } 2023-11-08 01:01:03,378 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 363), ProcessCount=165 (was 166), AvailableMemoryMB=8325 (was 8482)