2023-12-16 14:53:16,689 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-16 14:53:17,021 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-16 14:53:17,132 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-16 14:53:17,207 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-16 14:53:17,217 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-16 14:53:17,232 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=331, ProcessCount=171, AvailableMemoryMB=7411 2023-12-16 14:53:17,234 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-16 14:53:17,235 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-16 14:53:17,371 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013881715:84 , srv1020838969:84 , srv102956982:84 , srv1032199818:84 , srv10329161:84 , srv1049564074:84 , srv111316370:84 , srv1114551287:84 , srv1160285986:84 , srv1167051002:84 , srv1177851739:84 , srv1212157105:84 , srv1233704153:84 , srv1243412499:84 , srv1253202939:84 , srv1317532840:84 , srv1323715512:84 , srv132449167:84 , srv1343065833:84 , srv137354780:84 , srv1399896445:84 , srv1421292957:84 , srv1448789129:84 , srv1461453432:84 , srv1461702874:84 , srv1466205911:84 , srv1484368290:84 , srv1506821753:84 , srv1517809864:84 , srv1542152154:84 , srv1550316565:84 , srv1555579315:84 , srv1584646358:84 , srv1648348699:84 , srv166495396:84 , srv1685263901:84 , srv1715093202:84 , srv1759131300:84 , srv1780031922:84 , srv1794642131:84 , srv180005279:84 , srv1801787187:84 , srv1802401384:84 , srv1803651950:84 , srv1812043573:84 , srv1818241397:84 , srv1821153905:84 , srv1862214186:84 , srv1863556065:84 , srv1872740971:84 , srv1953435594:84 , srv1956892022:84 , srv1979649647:84 , srv1992512059:84 , srv2012174517:84 , srv2030146719:84 , srv2098801358:84 , srv2101144828:84 , srv2111898834:84 , srv2115726259:84 , srv249902114:84 , srv278650313:84 , srv278782561:84 , srv298929065:84 , srv304276355:84 , srv307401979:84 , srv325919429:84 , srv326141328:84 , srv334055959:84 , srv344500495:84 , srv400610894:84 , srv409917915:84 , srv416118935:84 , srv46887856:84 , srv47713159:84 , srv486079119:84 , srv501517894:84 , srv513552558:84 , srv515729089:84 , srv542633796:84 , srv5532076:84 , srv575677369:84 , srv580688512:84 , srv583205309:84 , srv600486911:84 , srv62868622:84 , srv633762642:84 , srv666476784:84 , srv672729003:84 , srv694703490:84 , srv774428905:84 , srv794528125:84 , srv801182996:84 , srv850856837:84 , srv887026664:84 , srv897407498:84 , srv943407366:84 , srv960327443:84 , srv979968860:84 , srv690292471:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-16 14:53:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 14:53:17,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv325919429=66, srv960327443=98, srv1956892022=51, srv344500495=69, srv2115726259=59, srv1780031922=38, srv486079119=75, srv278650313=61, srv1317532840=15, srv801182996=93, srv334055959=68, srv1555579315=31, srv307401979=65, srv1177851739=10, srv633762642=86, srv1013881715=0, srv46887856=73, srv2012174517=54, srv47713159=74, srv1343065833=18, srv850856837=94, srv1759131300=37, srv1049564074=5, srv1167051002=9, srv1114551287=7, srv1818241397=45, srv1466205911=25, srv1542152154=29, srv1517809864=28, srv979968860=99, srv137354780=19, srv774428905=91, srv794528125=92, srv1872740971=49, srv2030146719=55, srv111316370=6, srv1821153905=46, srv278782561=62, srv62868622=85, srv897407498=96, srv1803651950=43, srv666476784=87, srv501517894=76, srv1584646358=32, srv2101144828=57, srv132449167=17, srv690292471=89, srv304276355=64, srv513552558=77, srv887026664=95, srv672729003=88, srv1243412499=13, srv5532076=80, srv102956982=2, srv1648348699=33, srv1715093202=36, srv575677369=81, srv943407366=97, srv10329161=4, srv1461702874=24, srv1979649647=52, srv542633796=79, srv2111898834=58, srv1992512059=53, srv1421292957=21, srv166495396=34, srv416118935=72, srv1212157105=11, srv1484368290=26, srv249902114=60, srv2098801358=56, srv400610894=70, srv1160285986=8, srv1863556065=48, srv1506821753=27, srv298929065=63, srv1812043573=44, srv326141328=67, srv515729089=78, srv1032199818=3, srv580688512=82, srv1323715512=16, srv1448789129=22, srv1550316565=30, srv1801787187=41, srv583205309=83, srv694703490=90, srv1862214186=47, srv1802401384=42, srv1685263901=35, srv1020838969=1, srv1461453432=23, srv1233704153=12, srv409917915=71, srv1794642131=39, srv1253202939=14, srv180005279=40, srv1953435594=50, srv600486911=84, srv1399896445=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 14:53:17,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 14:53:17,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 14:53:17,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 14:53:17,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 14:53:17,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 14:53:17,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 14:53:17,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 14:53:17,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 14:53:17,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 14:53:17,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 14:53:17,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 14:53:17,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 14:53:17,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 14:53:17,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 14:53:17,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 14:53:17,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 14:53:17,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 14:53:17,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 14:53:17,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 14:53:17,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 14:53:17,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 14:53:17,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 14:53:17,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 14:53:17,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 14:53:17,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 14:53:17,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 14:53:17,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 14:53:17,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 14:53:17,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 14:53:17,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 14:53:17,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-16 14:53:17,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-16 14:53:17,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-16 14:53:17,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-16 14:53:17,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-16 14:53:17,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-16 14:53:17,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-16 14:53:17,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-16 14:53:17,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-16 14:53:17,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-16 14:53:17,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-16 14:53:17,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-16 14:53:17,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-16 14:53:17,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-16 14:53:17,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-16 14:53:17,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-16 14:53:17,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-16 14:53:17,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-16 14:53:17,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-16 14:53:17,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-16 14:53:17,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-16 14:53:17,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-16 14:53:17,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-16 14:53:17,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-16 14:53:17,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-16 14:53:17,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-16 14:53:17,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-16 14:53:17,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-16 14:53:17,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-16 14:53:17,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-16 14:53:17,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-16 14:53:17,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-16 14:53:17,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-16 14:53:17,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-16 14:53:17,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-16 14:53:17,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-16 14:53:17,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-16 14:53:17,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-16 14:53:17,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-16 14:53:17,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-16 14:53:17,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-16 14:53:17,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-16 14:53:17,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-16 14:53:17,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-16 14:53:17,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-16 14:53:17,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-16 14:53:17,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-16 14:53:17,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-16 14:53:17,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-16 14:53:17,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-16 14:53:17,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-16 14:53:17,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-16 14:53:17,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-16 14:53:17,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-16 14:53:17,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-16 14:53:17,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-16 14:53:17,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-16 14:53:17,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-16 14:53:17,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-16 14:53:17,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-16 14:53:17,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-16 14:53:17,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-16 14:53:17,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-16 14:53:17,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-16 14:53:17,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-16 14:53:17,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-16 14:53:17,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-16 14:53:17,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-16 14:53:17,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-16 14:53:17,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-16 14:53:17,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-16 14:53:17,874 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:53:17,874 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-16 14:53:17,874 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-16 14:53:18,266 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 733 ms to try 5000 different iterations. Found a solution that moves 4676 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18110485993496822. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04107940051627343, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05694031766273445, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5195555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04107940051627343, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17608333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24541666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:53:18,282 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv515729089:81 , srv1399896445:82 , srv1461453432:82 , srv1555579315:82 , srv166495396:82 , srv1862214186:82 , srv304276355:82 , srv694703490:82 , srv1013881715:83 , srv1160285986:83 , srv1421292957:83 , srv1685263901:83 , srv47713159:83 , srv501517894:83 , srv666476784:83 , srv960327443:83 , srv1114551287:84 , srv1780031922:84 , srv1794642131:84 , srv180005279:84 , srv344500495:84 , srv409917915:84 , srv542633796:84 , srv5532076:84 , srv583205309:84 , srv600486911:84 , srv887026664:84 , srv1323715512:85 , srv1803651950:85 , srv1812043573:85 , srv1953435594:85 , srv2012174517:85 , srv2098801358:85 , srv2101144828:85 , srv2111898834:85 , srv278782561:85 , srv298929065:85 , srv325919429:85 , srv633762642:85 , srv794528125:85 , srv102956982:86 , srv111316370:86 , srv132449167:86 , srv137354780:86 , srv1542152154:86 , srv1584646358:86 , srv1648348699:86 , srv1979649647:86 , srv2030146719:86 , srv416118935:86 , srv486079119:86 , srv580688512:86 , srv672729003:86 , srv801182996:86 , srv1032199818:87 , srv1212157105:87 , srv1253202939:87 , srv1484368290:87 , srv1506821753:87 , srv1550316565:87 , srv1715093202:87 , srv1818241397:87 , srv1863556065:87 , srv1956892022:87 , srv2115726259:87 , srv249902114:87 , srv278650313:87 , srv513552558:87 , srv943407366:87 , srv979968860:87 , srv1049564074:88 , srv1167051002:88 , srv1448789129:88 , srv1461702874:88 , srv1466205911:88 , srv1992512059:88 , srv307401979:88 , srv326141328:88 , srv62868622:88 , srv1177851739:89 , srv1233704153:89 , srv1517809864:89 , srv1821153905:89 , srv1872740971:89 , srv850856837:89 , srv1343065833:90 , srv1759131300:90 , srv1801787187:90 , srv1802401384:90 , srv334055959:90 , srv575677369:90 , srv774428905:90 , srv1243412499:91 , srv1317532840:91 , srv400610894:91 , srv46887856:92 , srv897407498:92 , srv10329161:93 , srv1020838969:95 , srv690292471:455 } 2023-12-16 14:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 14:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv325919429=66, srv960327443=98, srv1956892022=51, srv344500495=69, srv2115726259=59, srv1780031922=38, srv486079119=75, srv278650313=61, srv1317532840=15, srv801182996=93, srv334055959=68, srv1555579315=31, srv307401979=65, srv1177851739=10, srv633762642=86, srv1013881715=0, srv46887856=73, srv2012174517=54, srv47713159=74, srv1343065833=18, srv850856837=94, srv1759131300=37, srv1049564074=5, srv1167051002=9, srv1114551287=7, srv1818241397=45, srv1466205911=25, srv1542152154=29, srv1517809864=28, srv979968860=99, srv137354780=19, srv774428905=91, srv794528125=92, srv1872740971=49, srv2030146719=55, srv111316370=6, srv1821153905=46, srv278782561=62, srv62868622=85, srv897407498=96, srv1803651950=43, srv666476784=87, srv501517894=76, srv1584646358=32, srv2101144828=57, srv132449167=17, srv690292471=89, srv304276355=64, srv513552558=77, srv887026664=95, srv672729003=88, srv1243412499=13, srv5532076=80, srv102956982=2, srv1648348699=33, srv1715093202=36, srv575677369=81, srv943407366=97, srv10329161=4, srv1461702874=24, srv1979649647=52, srv542633796=79, srv2111898834=58, srv1992512059=53, srv1421292957=21, srv166495396=34, srv416118935=72, srv1212157105=11, srv1484368290=26, srv249902114=60, srv2098801358=56, srv400610894=70, srv1160285986=8, srv1863556065=48, srv1506821753=27, srv298929065=63, srv1812043573=44, srv326141328=67, srv515729089=78, srv1032199818=3, srv580688512=82, srv1323715512=16, srv1448789129=22, srv1550316565=30, srv1801787187=41, srv583205309=83, srv694703490=90, srv1862214186=47, srv1802401384=42, srv1685263901=35, srv1020838969=1, srv1461453432=23, srv1233704153=12, srv409917915=71, srv1794642131=39, srv1253202939=14, srv180005279=40, srv1953435594=50, srv600486911=84, srv1399896445=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 14:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 14:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 14:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 14:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 14:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 14:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 14:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 14:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 14:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 14:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 14:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 14:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 14:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 14:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 14:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 14:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 14:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 14:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 14:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 14:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 14:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 14:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 14:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 14:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 14:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 14:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 14:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 14:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 14:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 14:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 14:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-16 14:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-16 14:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-16 14:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-16 14:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-16 14:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-16 14:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-16 14:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-16 14:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-16 14:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-16 14:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-16 14:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-16 14:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-16 14:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-16 14:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-16 14:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-16 14:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-16 14:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-16 14:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-16 14:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-16 14:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-16 14:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-16 14:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-16 14:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-16 14:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-16 14:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-16 14:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-16 14:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-16 14:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-16 14:53:18,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-16 14:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-16 14:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-16 14:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-16 14:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-16 14:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-16 14:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-16 14:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-16 14:53:18,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-16 14:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-16 14:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-16 14:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-16 14:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-16 14:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-16 14:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-16 14:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-16 14:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-16 14:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-16 14:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-16 14:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-16 14:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-16 14:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-16 14:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-16 14:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-16 14:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-16 14:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-16 14:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-16 14:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-16 14:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-16 14:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-16 14:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-16 14:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-16 14:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-16 14:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-16 14:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-16 14:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-16 14:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-16 14:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-16 14:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-16 14:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-16 14:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-16 14:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-16 14:53:18,477 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04107940051627343, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05694031766273445, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04107940051627343, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17608333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24541666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:53:18,477 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-16 14:53:18,477 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18107211788315833, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04107940051627343, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05694031766273445, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04107940051627343, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17608333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24541666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-16 14:53:18,731 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 377 ms to try 5000 different iterations. Found a solution that moves 2609 regions; Going from a computed imbalance of 0.18107211788315833 to a new imbalance of 0.015145744029405907. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024280647009078005); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037982215697815416, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28988888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024280647009078005); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0465, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:53:18,744 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv10329161:78 , srv46887856:78 , srv1020838969:79 , srv1317532840:80 , srv400610894:80 , srv575677369:80 , srv897407498:80 , srv1801787187:81 , srv334055959:81 , srv774428905:81 , srv1343065833:82 , srv1517809864:82 , srv1802401384:82 , srv850856837:82 , srv1049564074:83 , srv1212157105:83 , srv1243412499:83 , srv1506821753:83 , srv1821153905:83 , srv307401979:83 , srv326141328:83 , srv102956982:84 , srv1167051002:84 , srv1177851739:84 , srv1253202939:84 , srv1448789129:84 , srv1461702874:84 , srv1542152154:84 , srv1818241397:84 , srv1863556065:84 , srv1872740971:84 , srv1956892022:84 , srv278650313:84 , srv1323715512:85 , srv132449167:85 , srv1466205911:85 , srv1550316565:85 , srv1584646358:85 , srv1715093202:85 , srv1759131300:85 , srv1979649647:85 , srv1992512059:85 , srv2115726259:85 , srv249902114:85 , srv278782561:85 , srv513552558:85 , srv943407366:85 , srv1032199818:86 , srv111316370:86 , srv1233704153:86 , srv1780031922:86 , srv325919429:86 , srv416118935:86 , srv486079119:86 , srv600486911:86 , srv62868622:86 , srv672729003:86 , srv801182996:86 , srv979968860:86 , srv1114551287:87 , srv1160285986:87 , srv137354780:87 , srv1421292957:87 , srv1484368290:87 , srv1803651950:87 , srv1812043573:87 , srv2012174517:87 , srv2030146719:87 , srv2098801358:87 , srv2101144828:87 , srv542633796:87 , srv5532076:87 , srv580688512:87 , srv794528125:87 , srv1648348699:88 , srv1685263901:88 , srv180005279:88 , srv1862214186:88 , srv1953435594:88 , srv2111898834:88 , srv298929065:88 , srv344500495:88 , srv583205309:88 , srv633762642:88 , srv666476784:88 , srv1013881715:89 , srv1399896445:89 , srv1555579315:89 , srv1794642131:89 , srv304276355:89 , srv501517894:89 , srv694703490:89 , srv887026664:89 , srv1461453432:90 , srv166495396:90 , srv409917915:90 , srv515729089:90 , srv960327443:91 , srv47713159:92 , srv690292471:535 } 2023-12-16 14:53:18,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 14:53:18,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv325919429=66, srv960327443=98, srv1956892022=51, srv344500495=69, srv2115726259=59, srv1780031922=38, srv486079119=75, srv278650313=61, srv1317532840=15, srv801182996=93, srv334055959=68, srv1555579315=31, srv307401979=65, srv1177851739=10, srv633762642=86, srv1013881715=0, srv46887856=73, srv2012174517=54, srv47713159=74, srv1343065833=18, srv850856837=94, srv1759131300=37, srv1049564074=5, srv1167051002=9, srv1114551287=7, srv1818241397=45, srv1466205911=25, srv1542152154=29, srv1517809864=28, srv979968860=99, srv137354780=19, srv774428905=91, srv794528125=92, srv1872740971=49, srv2030146719=55, srv111316370=6, srv1821153905=46, srv278782561=62, srv62868622=85, srv897407498=96, srv1803651950=43, srv666476784=87, srv501517894=76, srv1584646358=32, srv2101144828=57, srv132449167=17, srv690292471=89, srv304276355=64, srv513552558=77, srv887026664=95, srv672729003=88, srv1243412499=13, srv5532076=80, srv102956982=2, srv1648348699=33, srv1715093202=36, srv575677369=81, srv943407366=97, srv10329161=4, srv1461702874=24, srv1979649647=52, srv542633796=79, srv2111898834=58, srv1992512059=53, srv1421292957=21, srv166495396=34, srv416118935=72, srv1212157105=11, srv1484368290=26, srv249902114=60, srv2098801358=56, srv400610894=70, srv1160285986=8, srv1863556065=48, srv1506821753=27, srv298929065=63, srv1812043573=44, srv326141328=67, srv515729089=78, srv1032199818=3, srv580688512=82, srv1323715512=16, srv1448789129=22, srv1550316565=30, srv1801787187=41, srv583205309=83, srv694703490=90, srv1862214186=47, srv1802401384=42, srv1685263901=35, srv1020838969=1, srv1461453432=23, srv1233704153=12, srv409917915=71, srv1794642131=39, srv1253202939=14, srv180005279=40, srv1953435594=50, srv600486911=84, srv1399896445=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 14:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 14:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 14:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 14:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 14:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 14:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 14:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 14:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-16 14:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-16 14:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-16 14:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-16 14:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-16 14:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-16 14:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-16 14:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-16 14:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-16 14:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-16 14:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-16 14:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-16 14:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-16 14:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-16 14:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-16 14:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-16 14:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-16 14:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-16 14:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-16 14:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-16 14:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-16 14:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-16 14:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-16 14:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-16 14:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-16 14:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-16 14:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-16 14:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-16 14:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-16 14:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-16 14:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-16 14:53:18,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-16 14:53:18,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-16 14:53:18,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-16 14:53:18,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-16 14:53:18,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-16 14:53:18,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-16 14:53:18,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-16 14:53:18,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-16 14:53:18,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-16 14:53:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-16 14:53:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-16 14:53:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-16 14:53:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-16 14:53:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-16 14:53:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-16 14:53:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-16 14:53:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-16 14:53:18,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-16 14:53:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-16 14:53:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-16 14:53:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-16 14:53:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-16 14:53:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-16 14:53:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-16 14:53:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-16 14:53:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-16 14:53:18,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-16 14:53:18,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-16 14:53:18,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-16 14:53:18,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-16 14:53:18,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-16 14:53:18,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-16 14:53:18,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-16 14:53:18,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-16 14:53:18,920 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024280647009078005); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037982215697815416, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024280647009078005); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0465, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:53:18,920 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-16 14:53:18,920 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01512747542094311, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024280647009078005); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037982215697815416, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024280647009078005); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0465, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-16 14:53:19,148 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 342 ms to try 5000 different iterations. Found a solution that moves 963 regions; Going from a computed imbalance of 0.01512747542094311 to a new imbalance of 8.12536862174536E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006995268562901327); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009858710050550758); MoveCostFunction : (multiplier=7.0, imbalance=0.107, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006995268562901327); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:53:19,150 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1020838969:83 , srv1233704153:83 , srv1243412499:83 , srv1466205911:83 , srv1648348699:83 , srv1759131300:83 , srv47713159:83 , srv62868622:83 , srv979968860:83 , srv1032199818:84 , srv1167051002:84 , srv1177851739:84 , srv137354780:84 , srv1484368290:84 , srv1872740971:84 , srv2098801358:84 , srv409917915:84 , srv513552558:84 , srv580688512:84 , srv633762642:84 , srv960327443:84 , srv1013881715:85 , srv1343065833:85 , srv1448789129:85 , srv1461453432:85 , srv166495396:85 , srv1715093202:85 , srv1794642131:85 , srv1812043573:85 , srv1821153905:85 , srv1953435594:85 , srv1992512059:85 , srv2012174517:85 , srv2030146719:85 , srv2101144828:85 , srv2111898834:85 , srv2115726259:85 , srv278650313:85 , srv298929065:85 , srv344500495:85 , srv416118935:85 , srv501517894:85 , srv672729003:85 , srv887026664:85 , srv897407498:85 , srv943407366:85 , srv111316370:86 , srv1253202939:86 , srv1317532840:86 , srv132449167:86 , srv1399896445:86 , srv1461702874:86 , srv1517809864:86 , srv1550316565:86 , srv1555579315:86 , srv1584646358:86 , srv1685263901:86 , srv180005279:86 , srv1801787187:86 , srv1802401384:86 , srv1803651950:86 , srv1818241397:86 , srv1862214186:86 , srv1956892022:86 , srv1979649647:86 , srv249902114:86 , srv304276355:86 , srv325919429:86 , srv334055959:86 , srv486079119:86 , srv515729089:86 , srv583205309:86 , srv666476784:86 , srv774428905:86 , srv801182996:86 , srv850856837:86 , srv10329161:87 , srv1049564074:87 , srv1114551287:87 , srv1160285986:87 , srv1421292957:87 , srv1542152154:87 , srv1863556065:87 , srv307401979:87 , srv326141328:87 , srv400610894:87 , srv542633796:87 , srv5532076:87 , srv600486911:87 , srv694703490:87 , srv794528125:87 , srv102956982:88 , srv1212157105:88 , srv1323715512:88 , srv1506821753:88 , srv1780031922:88 , srv46887856:88 , srv575677369:88 , srv278782561:89 , srv690292471:530 } 2023-12-16 14:53:19,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 14:53:19,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv325919429=66, srv960327443=98, srv1956892022=51, srv344500495=69, srv2115726259=59, srv1780031922=38, srv486079119=75, srv278650313=61, srv1317532840=15, srv801182996=93, srv334055959=68, srv1555579315=31, srv307401979=65, srv1177851739=10, srv633762642=86, srv1013881715=0, srv46887856=73, srv2012174517=54, srv47713159=74, srv1343065833=18, srv850856837=94, srv1759131300=37, srv1049564074=5, srv1167051002=9, srv1114551287=7, srv1818241397=45, srv1466205911=25, srv1542152154=29, srv1517809864=28, srv979968860=99, srv137354780=19, srv774428905=91, srv794528125=92, srv1872740971=49, srv2030146719=55, srv111316370=6, srv1821153905=46, srv278782561=62, srv62868622=85, srv897407498=96, srv1803651950=43, srv666476784=87, srv501517894=76, srv1584646358=32, srv2101144828=57, srv132449167=17, srv690292471=89, srv304276355=64, srv513552558=77, srv887026664=95, srv672729003=88, srv1243412499=13, srv5532076=80, srv102956982=2, srv1648348699=33, srv1715093202=36, srv575677369=81, srv943407366=97, srv10329161=4, srv1461702874=24, srv1979649647=52, srv542633796=79, srv2111898834=58, srv1992512059=53, srv1421292957=21, srv166495396=34, srv416118935=72, srv1212157105=11, srv1484368290=26, srv249902114=60, srv2098801358=56, srv400610894=70, srv1160285986=8, srv1863556065=48, srv1506821753=27, srv298929065=63, srv1812043573=44, srv326141328=67, srv515729089=78, srv1032199818=3, srv580688512=82, srv1323715512=16, srv1448789129=22, srv1550316565=30, srv1801787187=41, srv583205309=83, srv694703490=90, srv1862214186=47, srv1802401384=42, srv1685263901=35, srv1020838969=1, srv1461453432=23, srv1233704153=12, srv409917915=71, srv1794642131=39, srv1253202939=14, srv180005279=40, srv1953435594=50, srv600486911=84, srv1399896445=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 14:53:19,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 14:53:19,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 14:53:19,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 14:53:19,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 14:53:19,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 14:53:19,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 14:53:19,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 14:53:19,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 14:53:19,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 14:53:19,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 14:53:19,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 14:53:19,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 14:53:19,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 14:53:19,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 14:53:19,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 14:53:19,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-16 14:53:19,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-16 14:53:19,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-16 14:53:19,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-16 14:53:19,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-16 14:53:19,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-16 14:53:19,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-16 14:53:19,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-16 14:53:19,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-16 14:53:19,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-16 14:53:19,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-16 14:53:19,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-16 14:53:19,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-16 14:53:19,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-16 14:53:19,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-16 14:53:19,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-16 14:53:19,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-16 14:53:19,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-16 14:53:19,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-16 14:53:19,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-16 14:53:19,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-16 14:53:19,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-16 14:53:19,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-16 14:53:19,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-16 14:53:19,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-16 14:53:19,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-16 14:53:19,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-16 14:53:19,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-16 14:53:19,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-16 14:53:19,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-16 14:53:19,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-16 14:53:19,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-16 14:53:19,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-16 14:53:19,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-16 14:53:19,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-16 14:53:19,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-16 14:53:19,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-16 14:53:19,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-16 14:53:19,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-16 14:53:19,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-16 14:53:19,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-16 14:53:19,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-16 14:53:19,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-16 14:53:19,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-16 14:53:19,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-16 14:53:19,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-16 14:53:19,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-16 14:53:19,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-16 14:53:19,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-16 14:53:19,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-16 14:53:19,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-16 14:53:19,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-16 14:53:19,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-16 14:53:19,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-16 14:53:19,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-16 14:53:19,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-16 14:53:19,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-16 14:53:19,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-16 14:53:19,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-16 14:53:19,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-16 14:53:19,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-16 14:53:19,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-16 14:53:19,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-16 14:53:19,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-16 14:53:19,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-16 14:53:19,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-16 14:53:19,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-16 14:53:19,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-16 14:53:19,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-16 14:53:19,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-16 14:53:19,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-16 14:53:19,287 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006995268562901327); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009858710050550758); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006995268562901327); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:53:19,287 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-16 14:53:19,287 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.057937920520083E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006995268562901327); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009858710050550758); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006995268562901327); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-16 14:53:19,623 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 435 ms to try 5000 different iterations. Found a solution that moves 287 regions; Going from a computed imbalance of 8.057937920520083E-4 to a new imbalance of 3.450870585838988E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.03188888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:53:19,624 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1759131300:83 , srv278782561:83 , srv46887856:83 , srv794528125:83 , srv102956982:84 , srv10329161:84 , srv1049564074:84 , srv1212157105:84 , srv1243412499:84 , srv1323715512:84 , srv1461702874:84 , srv1484368290:84 , srv1506821753:84 , srv1550316565:84 , srv1648348699:84 , srv1780031922:84 , srv1794642131:84 , srv180005279:84 , srv1802401384:84 , srv1803651950:84 , srv1818241397:84 , srv1863556065:84 , srv1953435594:84 , srv1992512059:84 , srv2030146719:84 , srv2111898834:84 , srv249902114:84 , srv298929065:84 , srv307401979:84 , srv326141328:84 , srv400610894:84 , srv409917915:84 , srv47713159:84 , srv486079119:84 , srv542633796:84 , srv5532076:84 , srv575677369:84 , srv583205309:84 , srv694703490:84 , srv887026664:84 , srv960327443:84 , srv1013881715:85 , srv1020838969:85 , srv1032199818:85 , srv111316370:85 , srv1114551287:85 , srv1160285986:85 , srv1177851739:85 , srv1233704153:85 , srv1253202939:85 , srv1317532840:85 , srv132449167:85 , srv1343065833:85 , srv137354780:85 , srv1399896445:85 , srv1421292957:85 , srv1448789129:85 , srv1461453432:85 , srv1466205911:85 , srv1517809864:85 , srv1542152154:85 , srv1555579315:85 , srv166495396:85 , srv1685263901:85 , srv1715093202:85 , srv1801787187:85 , srv1812043573:85 , srv1821153905:85 , srv1872740971:85 , srv1956892022:85 , srv1979649647:85 , srv2012174517:85 , srv2098801358:85 , srv2101144828:85 , srv304276355:85 , srv325919429:85 , srv334055959:85 , srv344500495:85 , srv416118935:85 , srv515729089:85 , srv580688512:85 , srv600486911:85 , srv62868622:85 , srv633762642:85 , srv666476784:85 , srv672729003:85 , srv774428905:85 , srv801182996:85 , srv850856837:85 , srv897407498:85 , srv943407366:85 , srv1167051002:86 , srv1584646358:86 , srv1862214186:86 , srv2115726259:86 , srv278650313:86 , srv501517894:86 , srv513552558:86 , srv979968860:86 , srv690292471:622 } 2023-12-16 14:53:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 14:53:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv325919429=66, srv960327443=98, srv1956892022=51, srv344500495=69, srv2115726259=59, srv1780031922=38, srv486079119=75, srv278650313=61, srv1317532840=15, srv801182996=93, srv334055959=68, srv1555579315=31, srv307401979=65, srv1177851739=10, srv633762642=86, srv1013881715=0, srv46887856=73, srv2012174517=54, srv47713159=74, srv1343065833=18, srv850856837=94, srv1759131300=37, srv1049564074=5, srv1167051002=9, srv1114551287=7, srv1818241397=45, srv1466205911=25, srv1542152154=29, srv1517809864=28, srv979968860=99, srv137354780=19, srv774428905=91, srv794528125=92, srv1872740971=49, srv2030146719=55, srv111316370=6, srv1821153905=46, srv278782561=62, srv62868622=85, srv897407498=96, srv1803651950=43, srv666476784=87, srv501517894=76, srv1584646358=32, srv2101144828=57, srv132449167=17, srv690292471=89, srv304276355=64, srv513552558=77, srv887026664=95, srv672729003=88, srv1243412499=13, srv5532076=80, srv102956982=2, srv1648348699=33, srv1715093202=36, srv575677369=81, srv943407366=97, srv10329161=4, srv1461702874=24, srv1979649647=52, srv542633796=79, srv2111898834=58, srv1992512059=53, srv1421292957=21, srv166495396=34, srv416118935=72, srv1212157105=11, srv1484368290=26, srv249902114=60, srv2098801358=56, srv400610894=70, srv1160285986=8, srv1863556065=48, srv1506821753=27, srv298929065=63, srv1812043573=44, srv326141328=67, srv515729089=78, srv1032199818=3, srv580688512=82, srv1323715512=16, srv1448789129=22, srv1550316565=30, srv1801787187=41, srv583205309=83, srv694703490=90, srv1862214186=47, srv1802401384=42, srv1685263901=35, srv1020838969=1, srv1461453432=23, srv1233704153=12, srv409917915=71, srv1794642131=39, srv1253202939=14, srv180005279=40, srv1953435594=50, srv600486911=84, srv1399896445=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 14:53:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 14:53:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-16 14:53:19,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-16 14:53:19,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-16 14:53:19,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-16 14:53:19,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-16 14:53:19,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-16 14:53:19,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-16 14:53:19,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-16 14:53:19,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-16 14:53:19,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-16 14:53:19,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-16 14:53:19,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-16 14:53:19,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-16 14:53:19,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-16 14:53:19,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-16 14:53:19,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-16 14:53:19,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-16 14:53:19,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-16 14:53:19,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-16 14:53:19,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-16 14:53:19,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-16 14:53:19,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-16 14:53:19,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-16 14:53:19,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-16 14:53:19,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-16 14:53:19,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-16 14:53:19,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-16 14:53:19,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-16 14:53:19,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-16 14:53:19,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-16 14:53:19,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-16 14:53:19,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-16 14:53:19,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-16 14:53:19,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-16 14:53:19,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-16 14:53:19,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-16 14:53:19,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-16 14:53:19,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-16 14:53:19,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-16 14:53:19,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-16 14:53:19,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-16 14:53:19,753 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:53:19,753 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-16 14:53:19,754 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.2499088905985494E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-16 14:53:19,972 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 316 ms to try 5000 different iterations. Found a solution that moves 26 regions; Going from a computed imbalance of 3.2499088905985494E-5 to a new imbalance of 6.580041563428285E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0028888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:53:19,972 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv111316370:83 , srv1114551287:83 , srv1233704153:83 , srv1584646358:83 , srv2115726259:83 , srv501517894:83 , srv801182996:83 , srv1013881715:84 , srv1020838969:84 , srv102956982:84 , srv1032199818:84 , srv10329161:84 , srv1049564074:84 , srv1160285986:84 , srv1167051002:84 , srv1177851739:84 , srv1212157105:84 , srv1253202939:84 , srv1317532840:84 , srv1323715512:84 , srv132449167:84 , srv1343065833:84 , srv137354780:84 , srv1399896445:84 , srv1421292957:84 , srv1448789129:84 , srv1461453432:84 , srv1461702874:84 , srv1484368290:84 , srv1506821753:84 , srv1517809864:84 , srv1542152154:84 , srv1550316565:84 , srv1555579315:84 , srv166495396:84 , srv1685263901:84 , srv1715093202:84 , srv1780031922:84 , srv1794642131:84 , srv180005279:84 , srv1801787187:84 , srv1802401384:84 , srv1803651950:84 , srv1812043573:84 , srv1821153905:84 , srv1862214186:84 , srv1863556065:84 , srv1872740971:84 , srv1953435594:84 , srv1956892022:84 , srv1979649647:84 , srv1992512059:84 , srv2012174517:84 , srv2030146719:84 , srv2101144828:84 , srv2111898834:84 , srv249902114:84 , srv278650313:84 , srv278782561:84 , srv298929065:84 , srv304276355:84 , srv307401979:84 , srv325919429:84 , srv326141328:84 , srv334055959:84 , srv344500495:84 , srv400610894:84 , srv409917915:84 , srv416118935:84 , srv47713159:84 , srv486079119:84 , srv513552558:84 , srv515729089:84 , srv542633796:84 , srv5532076:84 , srv575677369:84 , srv580688512:84 , srv583205309:84 , srv600486911:84 , srv62868622:84 , srv633762642:84 , srv666476784:84 , srv672729003:84 , srv694703490:84 , srv774428905:84 , srv794528125:84 , srv850856837:84 , srv887026664:84 , srv897407498:84 , srv943407366:84 , srv960327443:84 , srv979968860:84 , srv1243412499:85 , srv1466205911:85 , srv1648348699:85 , srv1759131300:85 , srv1818241397:85 , srv2098801358:85 , srv46887856:85 , srv690292471:684 } 2023-12-16 14:53:19,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 14:53:19,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv325919429=66, srv960327443=98, srv1956892022=51, srv344500495=69, srv2115726259=59, srv1780031922=38, srv486079119=75, srv278650313=61, srv1317532840=15, srv801182996=93, srv334055959=68, srv1555579315=31, srv307401979=65, srv1177851739=10, srv633762642=86, srv1013881715=0, srv46887856=73, srv2012174517=54, srv47713159=74, srv1343065833=18, srv850856837=94, srv1759131300=37, srv1049564074=5, srv1167051002=9, srv1114551287=7, srv1818241397=45, srv1466205911=25, srv1542152154=29, srv1517809864=28, srv979968860=99, srv137354780=19, srv774428905=91, srv794528125=92, srv1872740971=49, srv2030146719=55, srv111316370=6, srv1821153905=46, srv278782561=62, srv62868622=85, srv897407498=96, srv1803651950=43, srv666476784=87, srv501517894=76, srv1584646358=32, srv2101144828=57, srv132449167=17, srv690292471=89, srv304276355=64, srv513552558=77, srv887026664=95, srv672729003=88, srv1243412499=13, srv5532076=80, srv102956982=2, srv1648348699=33, srv1715093202=36, srv575677369=81, srv943407366=97, srv10329161=4, srv1461702874=24, srv1979649647=52, srv542633796=79, srv2111898834=58, srv1992512059=53, srv1421292957=21, srv166495396=34, srv416118935=72, srv1212157105=11, srv1484368290=26, srv249902114=60, srv2098801358=56, srv400610894=70, srv1160285986=8, srv1863556065=48, srv1506821753=27, srv298929065=63, srv1812043573=44, srv326141328=67, srv515729089=78, srv1032199818=3, srv580688512=82, srv1323715512=16, srv1448789129=22, srv1550316565=30, srv1801787187=41, srv583205309=83, srv694703490=90, srv1862214186=47, srv1802401384=42, srv1685263901=35, srv1020838969=1, srv1461453432=23, srv1233704153=12, srv409917915=71, srv1794642131=39, srv1253202939=14, srv180005279=40, srv1953435594=50, srv600486911=84, srv1399896445=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 14:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 14:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 14:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 14:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 14:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 14:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 14:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-16 14:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-16 14:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-16 14:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-16 14:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-16 14:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-16 14:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-16 14:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-16 14:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-16 14:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-16 14:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-16 14:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-16 14:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-16 14:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-16 14:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-16 14:53:20,105 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:53:20,105 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv111316370:83 , srv1114551287:83 , srv1233704153:83 , srv1584646358:83 , srv2115726259:83 , srv501517894:83 , srv801182996:83 , srv1013881715:84 , srv1020838969:84 , srv102956982:84 , srv1032199818:84 , srv10329161:84 , srv1049564074:84 , srv1160285986:84 , srv1167051002:84 , srv1177851739:84 , srv1212157105:84 , srv1253202939:84 , srv1317532840:84 , srv1323715512:84 , srv132449167:84 , srv1343065833:84 , srv137354780:84 , srv1399896445:84 , srv1421292957:84 , srv1448789129:84 , srv1461453432:84 , srv1461702874:84 , srv1484368290:84 , srv1506821753:84 , srv1517809864:84 , srv1542152154:84 , srv1550316565:84 , srv1555579315:84 , srv166495396:84 , srv1685263901:84 , srv1715093202:84 , srv1780031922:84 , srv1794642131:84 , srv180005279:84 , srv1801787187:84 , srv1802401384:84 , srv1803651950:84 , srv1812043573:84 , srv1821153905:84 , srv1862214186:84 , srv1863556065:84 , srv1872740971:84 , srv1953435594:84 , srv1956892022:84 , srv1979649647:84 , srv1992512059:84 , srv2012174517:84 , srv2030146719:84 , srv2101144828:84 , srv2111898834:84 , srv249902114:84 , srv278650313:84 , srv278782561:84 , srv298929065:84 , srv304276355:84 , srv307401979:84 , srv325919429:84 , srv326141328:84 , srv334055959:84 , srv344500495:84 , srv400610894:84 , srv409917915:84 , srv416118935:84 , srv47713159:84 , srv486079119:84 , srv513552558:84 , srv515729089:84 , srv542633796:84 , srv5532076:84 , srv575677369:84 , srv580688512:84 , srv583205309:84 , srv600486911:84 , srv62868622:84 , srv633762642:84 , srv666476784:84 , srv672729003:84 , srv694703490:84 , srv774428905:84 , srv794528125:84 , srv850856837:84 , srv887026664:84 , srv897407498:84 , srv943407366:84 , srv960327443:84 , srv979968860:84 , srv1243412499:85 , srv1466205911:85 , srv1648348699:85 , srv1759131300:85 , srv1818241397:85 , srv2098801358:85 , srv46887856:85 , srv690292471:684 } 2023-12-16 14:53:20,129 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=336 (was 331) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=6813 (was 7411) 2023-12-16 14:53:20,136 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=336, ProcessCount=171, AvailableMemoryMB=6812 2023-12-16 14:53:20,137 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-16 14:53:20,137 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-16 14:53:20,152 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1092732854:84 , srv1099402130:84 , srv1234063367:84 , srv1384551522:84 , srv1403311935:84 , srv1413527162:84 , srv1423078847:84 , srv1685104371:84 , srv1731824707:84 , srv1754622113:84 , srv1858839735:84 , srv1873784432:84 , srv190619857:84 , srv1987940075:84 , srv2004951501:84 , srv2100084479:84 , srv2118939149:84 , srv2120754300:84 , srv357009576:84 , srv50820705:84 , srv556589130:84 , srv581038688:84 , srv671452380:84 , srv718121681:84 , srv776322846:84 , srv790133680:84 , srv826822653:84 , srv870556331:84 , srv903487651:84 , srv1418192368:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-16 14:53:20,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 14:53:20,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1731824707=9, srv1234063367=2, srv2120754300=18, srv1418192368=6, srv1987940075=14, srv870556331=28, srv1092732854=0, srv1384551522=3, srv1873784432=12, srv581038688=22, srv718121681=24, srv903487651=29, srv1423078847=7, srv1754622113=10, srv1858839735=11, srv2118939149=17, srv357009576=19, srv776322846=25, srv2004951501=15, srv826822653=27, srv1099402130=1, srv2100084479=16, srv671452380=23, srv556589130=21, srv790133680=26, srv50820705=20, srv190619857=13, srv1413527162=5, srv1685104371=8, srv1403311935=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 14:53:20,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 14:53:20,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 14:53:20,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 14:53:20,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 14:53:20,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 14:53:20,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 14:53:20,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 14:53:20,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 14:53:20,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 14:53:20,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 14:53:20,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 14:53:20,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 14:53:20,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-16 14:53:20,179 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1057602082247577, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:53:20,179 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912526290227007, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1057602082247577, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-16 14:55:20,166 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6161084 different iterations. Found a solution that moves 1856 regions; Going from a computed imbalance of 0.9912526290227007 to a new imbalance of 4.3319965896196796E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6874074074074074, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:55:20,174 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1092732854:90 , srv1099402130:90 , srv1234063367:90 , srv1384551522:90 , srv1403311935:90 , srv1413527162:90 , srv1418192368:90 , srv1423078847:90 , srv1685104371:90 , srv1731824707:90 , srv1754622113:90 , srv1858839735:90 , srv1873784432:90 , srv190619857:90 , srv1987940075:90 , srv2004951501:90 , srv2100084479:90 , srv2118939149:90 , srv2120754300:90 , srv357009576:90 , srv50820705:90 , srv556589130:90 , srv581038688:90 , srv671452380:90 , srv718121681:90 , srv776322846:90 , srv790133680:90 , srv826822653:90 , srv870556331:90 , srv903487651:90 } 2023-12-16 14:55:20,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 14:55:20,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1731824707=9, srv1234063367=2, srv2120754300=18, srv1418192368=6, srv1987940075=14, srv870556331=28, srv1092732854=0, srv1384551522=3, srv1873784432=12, srv581038688=22, srv718121681=24, srv903487651=29, srv1423078847=7, srv1754622113=10, srv1858839735=11, srv2118939149=17, srv357009576=19, srv776322846=25, srv2004951501=15, srv826822653=27, srv1099402130=1, srv2100084479=16, srv671452380=23, srv556589130=21, srv790133680=26, srv50820705=20, srv190619857=13, srv1413527162=5, srv1685104371=8, srv1403311935=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 14:55:20,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 14:55:20,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 14:55:20,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 14:55:20,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 14:55:20,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 14:55:20,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 14:55:20,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 14:55:20,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 14:55:20,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 14:55:20,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 14:55:20,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-16 14:55:20,196 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 14:55:20,196 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1092732854:90 , srv1099402130:90 , srv1234063367:90 , srv1384551522:90 , srv1403311935:90 , srv1413527162:90 , srv1418192368:90 , srv1423078847:90 , srv1685104371:90 , srv1731824707:90 , srv1754622113:90 , srv1858839735:90 , srv1873784432:90 , srv190619857:90 , srv1987940075:90 , srv2004951501:90 , srv2100084479:90 , srv2118939149:90 , srv2120754300:90 , srv357009576:90 , srv50820705:90 , srv556589130:90 , srv581038688:90 , srv671452380:90 , srv718121681:90 , srv776322846:90 , srv790133680:90 , srv826822653:90 , srv870556331:90 , srv903487651:90 } 2023-12-16 14:55:20,208 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=206 (was 336), ProcessCount=167 (was 171), AvailableMemoryMB=5769 (was 6812)