2023-04-11 23:55:44,381 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-04-11 23:55:44,788 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-11 23:55:44,922 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-11 23:55:44,988 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-11 23:55:44,996 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-11 23:55:45,010 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=361, ProcessCount=169, AvailableMemoryMB=1643 2023-04-11 23:55:45,013 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-11 23:55:45,013 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-11 23:55:45,175 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1002011996:84 , srv1019539513:84 , srv1030992496:84 , srv1045143294:84 , srv1084987072:84 , srv1114427149:84 , srv1121113790:84 , srv1133386197:84 , srv1155785667:84 , srv1189181598:84 , srv1247720753:84 , srv1256115793:84 , srv1259716198:84 , srv1270254316:84 , srv1285538288:84 , srv1316199073:84 , srv1317815967:84 , srv1324583018:84 , srv1339971715:84 , srv1354812701:84 , srv1357364179:84 , srv1375807515:84 , srv1409314330:84 , srv1424456934:84 , srv1427648031:84 , srv1436576460:84 , srv1439836670:84 , srv148607108:84 , srv1488510634:84 , srv1494824752:84 , srv1514247213:84 , srv1521844820:84 , srv1522936187:84 , srv1544476851:84 , srv1559523761:84 , srv1578564626:84 , srv1592459843:84 , srv1619237136:84 , srv1645817243:84 , srv165499731:84 , srv1695440458:84 , srv1710362324:84 , srv1723221008:84 , srv1827386453:84 , srv1835122455:84 , srv1875182482:84 , srv188467032:84 , srv1915951514:84 , srv1921035903:84 , srv1924962678:84 , srv1934479171:84 , srv1942560380:84 , srv1948164000:84 , srv1956422434:84 , srv2072927542:84 , srv2083539047:84 , srv2085775999:84 , srv2114469351:84 , srv2132781216:84 , srv214943485:84 , srv2713158:84 , srv279105715:84 , srv304983477:84 , srv389110227:84 , srv397352271:84 , srv426781502:84 , srv43116797:84 , srv44049629:84 , srv451924657:84 , srv462527659:84 , srv472977346:84 , srv510818620:84 , srv526658146:84 , srv528049723:84 , srv567497604:84 , srv570971846:84 , srv60495680:84 , srv62612927:84 , srv678094308:84 , srv733434374:84 , srv764993443:84 , srv776360013:84 , srv790855607:84 , srv802620233:84 , srv813337396:84 , srv833631637:84 , srv837003923:84 , srv869923274:84 , srv878943204:84 , srv880851940:84 , srv88820992:84 , srv915583672:84 , srv919479903:84 , srv920990511:84 , srv926454917:84 , srv942591694:84 , srv958715051:84 , srv980823327:84 , srv98199212:84 , srv1704935445:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-04-11 23:55:45,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-11 23:55:45,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1921035903=49, srv1121113790=6, srv60495680=77, srv1522936187=32, srv1427648031=24, srv472977346=71, srv426781502=66, srv1317815967=16, srv165499731=39, srv1875182482=46, srv1924962678=50, srv2085775999=57, srv528049723=74, srv1835122455=45, srv1494824752=29, srv980823327=98, srv148607108=27, srv1521844820=31, srv1578564626=35, srv1030992496=2, srv833631637=86, srv1357364179=20, srv678094308=79, srv813337396=85, srv1436576460=25, srv214943485=60, srv1544476851=33, srv389110227=64, srv1045143294=3, srv304983477=63, srv880851940=90, srv1915951514=48, srv2132781216=59, srv1424456934=23, srv1189181598=9, srv1375807515=21, srv1942560380=52, srv1247720753=10, srv1114427149=5, srv1948164000=53, srv1619237136=37, srv510818620=72, srv1514247213=30, srv2072927542=55, srv1316199073=15, srv919479903=93, srv958715051=97, srv188467032=47, srv1956422434=54, srv2114469351=58, srv1645817243=38, srv733434374=80, srv1704935445=41, srv462527659=70, srv2083539047=56, srv1155785667=8, srv1723221008=43, srv1019539513=1, srv1259716198=12, srv570971846=76, srv98199212=99, srv1339971715=18, srv1827386453=44, srv1270254316=13, srv1002011996=0, srv1285538288=14, srv1409314330=22, srv1439836670=26, srv279105715=62, srv920990511=94, srv526658146=73, srv44049629=68, srv1695440458=40, srv837003923=87, srv451924657=69, srv790855607=83, srv1133386197=7, srv88820992=91, srv926454917=95, srv62612927=78, srv1084987072=4, srv1934479171=51, srv567497604=75, srv942591694=96, srv1710362324=42, srv1592459843=36, srv802620233=84, srv2713158=61, srv764993443=81, srv1324583018=17, srv878943204=89, srv869923274=88, srv1256115793=11, srv1488510634=28, srv397352271=65, srv43116797=67, srv1354812701=19, srv915583672=92, srv1559523761=34, srv776360013=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-11 23:55:45,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-11 23:55:45,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-11 23:55:45,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-11 23:55:45,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-11 23:55:45,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-11 23:55:45,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-11 23:55:45,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-11 23:55:45,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-11 23:55:45,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-11 23:55:45,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-11 23:55:45,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-11 23:55:45,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-11 23:55:45,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-11 23:55:45,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-11 23:55:45,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-11 23:55:45,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-11 23:55:45,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-11 23:55:45,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-11 23:55:45,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-11 23:55:45,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-11 23:55:45,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-11 23:55:45,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-11 23:55:45,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-11 23:55:45,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-11 23:55:45,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-11 23:55:45,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-11 23:55:45,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-11 23:55:45,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-11 23:55:45,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-11 23:55:45,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-11 23:55:45,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-11 23:55:45,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-11 23:55:45,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-11 23:55:45,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-11 23:55:45,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-11 23:55:45,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-11 23:55:45,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-11 23:55:45,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-11 23:55:45,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-11 23:55:45,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-11 23:55:45,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-11 23:55:45,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-11 23:55:45,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-11 23:55:45,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-11 23:55:45,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-11 23:55:45,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-11 23:55:45,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-11 23:55:45,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-11 23:55:45,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-11 23:55:45,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-11 23:55:45,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-11 23:55:45,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-11 23:55:45,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-11 23:55:45,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-11 23:55:45,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-11 23:55:45,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-11 23:55:45,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-11 23:55:45,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-11 23:55:45,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-11 23:55:45,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-11 23:55:45,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-11 23:55:45,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-11 23:55:45,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-11 23:55:45,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-11 23:55:45,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-11 23:55:45,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-11 23:55:45,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-11 23:55:45,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-11 23:55:45,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-11 23:55:45,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-11 23:55:45,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-11 23:55:45,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-11 23:55:45,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-11 23:55:45,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-11 23:55:45,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-11 23:55:45,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-11 23:55:45,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-11 23:55:45,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-11 23:55:45,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-11 23:55:45,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-11 23:55:45,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-11 23:55:45,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-11 23:55:45,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-11 23:55:45,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-11 23:55:45,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-11 23:55:45,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-11 23:55:45,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-11 23:55:45,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-11 23:55:45,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-11 23:55:45,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-11 23:55:45,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-11 23:55:45,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-11 23:55:45,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-11 23:55:45,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-11 23:55:45,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-11 23:55:45,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-11 23:55:45,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-11 23:55:45,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-11 23:55:45,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-11 23:55:45,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-11 23:55:45,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-11 23:55:45,573 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:55:45,574 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-11 23:55:45,574 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-11 23:55:45,998 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 634 ms to try 5000 different iterations. Found a solution that moves 4697 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17800052672675806. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03627739492736557, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05573909077281018, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5218888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03627739492736557, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17316666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24041666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:55:46,034 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1695440458:81 , srv2085775999:81 , srv88820992:81 , srv1942560380:82 , srv915583672:82 , srv920990511:82 , srv1114427149:83 , srv1133386197:83 , srv1256115793:83 , srv1723221008:83 , srv1835122455:83 , srv188467032:83 , srv389110227:83 , srv1030992496:84 , srv2072927542:84 , srv2114469351:84 , srv426781502:84 , srv43116797:84 , srv528049723:84 , srv958715051:84 , srv1084987072:85 , srv1339971715:85 , srv1357364179:85 , srv1409314330:85 , srv1439836670:85 , srv1521844820:85 , srv1522936187:85 , srv2132781216:85 , srv214943485:85 , srv472977346:85 , srv764993443:85 , srv833631637:85 , srv837003923:85 , srv926454917:85 , srv1019539513:86 , srv1247720753:86 , srv1259716198:86 , srv1270254316:86 , srv1316199073:86 , srv1427648031:86 , srv1488510634:86 , srv1578564626:86 , srv1875182482:86 , srv397352271:86 , srv570971846:86 , srv62612927:86 , srv813337396:86 , srv1189181598:87 , srv1317815967:87 , srv1514247213:87 , srv1544476851:87 , srv1592459843:87 , srv1645817243:87 , srv1921035903:87 , srv279105715:87 , srv304983477:87 , srv44049629:87 , srv567497604:87 , srv678094308:87 , srv869923274:87 , srv878943204:87 , srv1002011996:88 , srv1121113790:88 , srv1375807515:88 , srv1424456934:88 , srv1559523761:88 , srv1956422434:88 , srv451924657:88 , srv510818620:88 , srv526658146:88 , srv790855607:88 , srv802620233:88 , srv919479903:88 , srv980823327:88 , srv1045143294:89 , srv1354812701:89 , srv1436576460:89 , srv1494824752:89 , srv1619237136:89 , srv1915951514:89 , srv60495680:89 , srv776360013:89 , srv880851940:89 , srv98199212:89 , srv1285538288:90 , srv165499731:90 , srv1710362324:90 , srv1934479171:90 , srv1948164000:90 , srv2083539047:90 , srv733434374:90 , srv942591694:90 , srv1155785667:91 , srv1924962678:91 , srv462527659:92 , srv1827386453:93 , srv2713158:93 , srv1324583018:95 , srv148607108:95 , srv1704935445:412 } 2023-04-11 23:55:46,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-11 23:55:46,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1921035903=49, srv1121113790=6, srv60495680=77, srv1522936187=32, srv1427648031=24, srv472977346=71, srv426781502=66, srv1317815967=16, srv165499731=39, srv1875182482=46, srv1924962678=50, srv2085775999=57, srv528049723=74, srv1835122455=45, srv1494824752=29, srv980823327=98, srv148607108=27, srv1521844820=31, srv1578564626=35, srv1030992496=2, srv833631637=86, srv1357364179=20, srv678094308=79, srv813337396=85, srv1436576460=25, srv214943485=60, srv1544476851=33, srv389110227=64, srv1045143294=3, srv304983477=63, srv880851940=90, srv1915951514=48, srv2132781216=59, srv1424456934=23, srv1189181598=9, srv1375807515=21, srv1942560380=52, srv1247720753=10, srv1114427149=5, srv1948164000=53, srv1619237136=37, srv510818620=72, srv1514247213=30, srv2072927542=55, srv1316199073=15, srv919479903=93, srv958715051=97, srv188467032=47, srv1956422434=54, srv2114469351=58, srv1645817243=38, srv733434374=80, srv1704935445=41, srv462527659=70, srv2083539047=56, srv1155785667=8, srv1723221008=43, srv1019539513=1, srv1259716198=12, srv570971846=76, srv98199212=99, srv1339971715=18, srv1827386453=44, srv1270254316=13, srv1002011996=0, srv1285538288=14, srv1409314330=22, srv1439836670=26, srv279105715=62, srv920990511=94, srv526658146=73, srv44049629=68, srv1695440458=40, srv837003923=87, srv451924657=69, srv790855607=83, srv1133386197=7, srv88820992=91, srv926454917=95, srv62612927=78, srv1084987072=4, srv1934479171=51, srv567497604=75, srv942591694=96, srv1710362324=42, srv1592459843=36, srv802620233=84, srv2713158=61, srv764993443=81, srv1324583018=17, srv878943204=89, srv869923274=88, srv1256115793=11, srv1488510634=28, srv397352271=65, srv43116797=67, srv1354812701=19, srv915583672=92, srv1559523761=34, srv776360013=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-11 23:55:46,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-11 23:55:46,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-11 23:55:46,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-11 23:55:46,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-11 23:55:46,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-11 23:55:46,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-11 23:55:46,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-11 23:55:46,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-11 23:55:46,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-11 23:55:46,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-11 23:55:46,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-11 23:55:46,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-11 23:55:46,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-11 23:55:46,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-11 23:55:46,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-11 23:55:46,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-11 23:55:46,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-11 23:55:46,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-11 23:55:46,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-11 23:55:46,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-11 23:55:46,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-11 23:55:46,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-11 23:55:46,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-11 23:55:46,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-11 23:55:46,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-11 23:55:46,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-11 23:55:46,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-11 23:55:46,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-11 23:55:46,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-11 23:55:46,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-11 23:55:46,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-11 23:55:46,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-11 23:55:46,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-11 23:55:46,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-11 23:55:46,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-11 23:55:46,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-11 23:55:46,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-11 23:55:46,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-11 23:55:46,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-11 23:55:46,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-11 23:55:46,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-11 23:55:46,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-11 23:55:46,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-11 23:55:46,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-11 23:55:46,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-11 23:55:46,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-11 23:55:46,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-11 23:55:46,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-11 23:55:46,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-11 23:55:46,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-11 23:55:46,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-11 23:55:46,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-11 23:55:46,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-11 23:55:46,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-11 23:55:46,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-11 23:55:46,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-11 23:55:46,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-11 23:55:46,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-11 23:55:46,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-11 23:55:46,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-11 23:55:46,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-11 23:55:46,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-11 23:55:46,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-11 23:55:46,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-11 23:55:46,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-11 23:55:46,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-11 23:55:46,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-11 23:55:46,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-11 23:55:46,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-11 23:55:46,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-11 23:55:46,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-11 23:55:46,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-11 23:55:46,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-11 23:55:46,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-11 23:55:46,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-11 23:55:46,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-11 23:55:46,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-11 23:55:46,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-11 23:55:46,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-11 23:55:46,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-11 23:55:46,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-11 23:55:46,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-11 23:55:46,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-11 23:55:46,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-11 23:55:46,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-11 23:55:46,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-11 23:55:46,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-11 23:55:46,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-11 23:55:46,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-11 23:55:46,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-11 23:55:46,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-11 23:55:46,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-11 23:55:46,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-11 23:55:46,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-11 23:55:46,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-11 23:55:46,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-11 23:55:46,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-11 23:55:46,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-11 23:55:46,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-11 23:55:46,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-11 23:55:46,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-11 23:55:46,269 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03627739492736557, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05573909077281018, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03627739492736557, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17316666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24041666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:55:46,269 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-11 23:55:46,270 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17796763762980528, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03627739492736557, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05573909077281018, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03627739492736557, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17316666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24041666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-11 23:55:46,654 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 540 ms to try 5000 different iterations. Found a solution that moves 2519 regions; Going from a computed imbalance of 0.17796763762980528 to a new imbalance of 0.015999332720764044. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019930669206259802); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03573816742013983, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2798888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019930669206259802); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04466666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:55:46,669 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1324583018:77 , srv148607108:77 , srv2713158:78 , srv1827386453:80 , srv1045143294:81 , srv1924962678:81 , srv1948164000:81 , srv462527659:81 , srv1121113790:82 , srv1155785667:82 , srv2083539047:82 , srv733434374:82 , srv942591694:82 , srv1002011996:83 , srv1375807515:83 , srv1424456934:83 , srv1592459843:83 , srv1710362324:83 , srv1915951514:83 , srv1921035903:83 , srv60495680:83 , srv776360013:83 , srv790855607:83 , srv98199212:83 , srv1285538288:84 , srv1317815967:84 , srv1494824752:84 , srv1559523761:84 , srv1619237136:84 , srv1645817243:84 , srv165499731:84 , srv1875182482:84 , srv510818620:84 , srv567497604:84 , srv802620233:84 , srv878943204:84 , srv1189181598:85 , srv1316199073:85 , srv1436576460:85 , srv1514247213:85 , srv1521844820:85 , srv1934479171:85 , srv1956422434:85 , srv397352271:85 , srv44049629:85 , srv451924657:85 , srv62612927:85 , srv833631637:85 , srv837003923:85 , srv869923274:85 , srv919479903:85 , srv980823327:85 , srv1354812701:86 , srv1357364179:86 , srv1427648031:86 , srv1522936187:86 , srv1544476851:86 , srv2114469351:86 , srv279105715:86 , srv304983477:86 , srv526658146:86 , srv528049723:86 , srv813337396:86 , srv926454917:86 , srv1247720753:87 , srv1259716198:87 , srv1270254316:87 , srv1409314330:87 , srv1578564626:87 , srv2072927542:87 , srv2132781216:87 , srv678094308:87 , srv958715051:87 , srv1019539513:88 , srv1439836670:88 , srv1488510634:88 , srv188467032:88 , srv214943485:88 , srv472977346:88 , srv570971846:88 , srv880851940:88 , srv1030992496:89 , srv1084987072:89 , srv1133386197:89 , srv1723221008:89 , srv1835122455:89 , srv389110227:89 , srv426781502:89 , srv43116797:89 , srv920990511:89 , srv1114427149:90 , srv1256115793:90 , srv1695440458:90 , srv2085775999:90 , srv764993443:90 , srv88820992:90 , srv915583672:90 , srv1942560380:91 , srv1339971715:93 , srv1704935445:539 } 2023-04-11 23:55:46,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-11 23:55:46,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1921035903=49, srv1121113790=6, srv60495680=77, srv1522936187=32, srv1427648031=24, srv472977346=71, srv426781502=66, srv1317815967=16, srv165499731=39, srv1875182482=46, srv1924962678=50, srv2085775999=57, srv528049723=74, srv1835122455=45, srv1494824752=29, srv980823327=98, srv148607108=27, srv1521844820=31, srv1578564626=35, srv1030992496=2, srv833631637=86, srv1357364179=20, srv678094308=79, srv813337396=85, srv1436576460=25, srv214943485=60, srv1544476851=33, srv389110227=64, srv1045143294=3, srv304983477=63, srv880851940=90, srv1915951514=48, srv2132781216=59, srv1424456934=23, srv1189181598=9, srv1375807515=21, srv1942560380=52, srv1247720753=10, srv1114427149=5, srv1948164000=53, srv1619237136=37, srv510818620=72, srv1514247213=30, srv2072927542=55, srv1316199073=15, srv919479903=93, srv958715051=97, srv188467032=47, srv1956422434=54, srv2114469351=58, srv1645817243=38, srv733434374=80, srv1704935445=41, srv462527659=70, srv2083539047=56, srv1155785667=8, srv1723221008=43, srv1019539513=1, srv1259716198=12, srv570971846=76, srv98199212=99, srv1339971715=18, srv1827386453=44, srv1270254316=13, srv1002011996=0, srv1285538288=14, srv1409314330=22, srv1439836670=26, srv279105715=62, srv920990511=94, srv526658146=73, srv44049629=68, srv1695440458=40, srv837003923=87, srv451924657=69, srv790855607=83, srv1133386197=7, srv88820992=91, srv926454917=95, srv62612927=78, srv1084987072=4, srv1934479171=51, srv567497604=75, srv942591694=96, srv1710362324=42, srv1592459843=36, srv802620233=84, srv2713158=61, srv764993443=81, srv1324583018=17, srv878943204=89, srv869923274=88, srv1256115793=11, srv1488510634=28, srv397352271=65, srv43116797=67, srv1354812701=19, srv915583672=92, srv1559523761=34, srv776360013=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-11 23:55:46,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-11 23:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-11 23:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-11 23:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-11 23:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-11 23:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-11 23:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-11 23:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-11 23:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-11 23:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-11 23:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-11 23:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-11 23:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-11 23:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-11 23:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-11 23:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-11 23:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-11 23:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-11 23:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-11 23:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-11 23:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-11 23:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-11 23:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-11 23:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-11 23:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-11 23:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-11 23:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-11 23:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-11 23:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-11 23:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-11 23:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-11 23:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-11 23:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-11 23:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-11 23:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-11 23:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-11 23:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-11 23:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-11 23:55:46,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-11 23:55:46,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-11 23:55:46,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-11 23:55:46,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-11 23:55:46,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-11 23:55:46,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-11 23:55:46,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-11 23:55:46,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-11 23:55:46,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-11 23:55:46,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-11 23:55:46,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-11 23:55:46,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-11 23:55:46,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-11 23:55:46,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-11 23:55:46,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-11 23:55:46,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-11 23:55:46,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-11 23:55:46,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-11 23:55:46,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-11 23:55:46,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-11 23:55:46,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-11 23:55:46,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-11 23:55:46,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-11 23:55:46,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-11 23:55:46,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-11 23:55:46,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-11 23:55:46,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-11 23:55:46,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-11 23:55:46,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-11 23:55:46,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-11 23:55:46,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-11 23:55:46,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-11 23:55:46,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-11 23:55:46,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-11 23:55:46,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-11 23:55:46,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-11 23:55:46,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-11 23:55:46,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-11 23:55:46,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-11 23:55:46,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-11 23:55:46,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-11 23:55:46,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-11 23:55:46,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-11 23:55:46,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-11 23:55:46,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-11 23:55:46,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-11 23:55:46,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-11 23:55:46,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-11 23:55:46,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-11 23:55:46,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-11 23:55:46,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-11 23:55:46,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-11 23:55:46,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-11 23:55:46,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-11 23:55:46,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-11 23:55:46,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-11 23:55:46,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-11 23:55:46,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-11 23:55:46,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-11 23:55:46,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-11 23:55:46,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-11 23:55:46,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-11 23:55:46,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-11 23:55:46,847 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019930669206259802); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03573816742013983, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019930669206259802); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04466666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:55:46,847 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-11 23:55:46,847 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015981694305770643, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019930669206259802); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03573816742013983, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019930669206259802); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04466666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-11 23:55:47,177 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 456 ms to try 5000 different iterations. Found a solution that moves 955 regions; Going from a computed imbalance of 0.015981694305770643 to a new imbalance of 6.556990726613261E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005650148677010918); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0098016238147137); MoveCostFunction : (multiplier=7.0, imbalance=0.10611111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005650148677010918); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:55:47,179 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1339971715:80 , srv880851940:81 , srv1354812701:82 , srv1084987072:83 , srv1155785667:83 , srv1285538288:83 , srv1436576460:83 , srv165499731:83 , srv1934479171:83 , srv526658146:83 , srv570971846:83 , srv764993443:83 , srv1114427149:84 , srv1256115793:84 , srv1270254316:84 , srv1488510634:84 , srv1494824752:84 , srv1544476851:84 , srv1578564626:84 , srv1827386453:84 , srv214943485:84 , srv304983477:84 , srv43116797:84 , srv451924657:84 , srv462527659:84 , srv678094308:84 , srv980823327:84 , srv1019539513:85 , srv1133386197:85 , srv1247720753:85 , srv1259716198:85 , srv1427648031:85 , srv1439836670:85 , srv1619237136:85 , srv1710362324:85 , srv1835122455:85 , srv1915951514:85 , srv1942560380:85 , srv1956422434:85 , srv2083539047:85 , srv279105715:85 , srv389110227:85 , srv426781502:85 , srv472977346:85 , srv510818620:85 , srv60495680:85 , srv776360013:85 , srv869923274:85 , srv919479903:85 , srv942591694:85 , srv1002011996:86 , srv1030992496:86 , srv1189181598:86 , srv1316199073:86 , srv1317815967:86 , srv1324583018:86 , srv1409314330:86 , srv148607108:86 , srv1514247213:86 , srv1522936187:86 , srv1559523761:86 , srv1723221008:86 , srv1924962678:86 , srv1948164000:86 , srv2072927542:86 , srv2132781216:86 , srv2713158:86 , srv397352271:86 , srv44049629:86 , srv62612927:86 , srv733434374:86 , srv802620233:86 , srv837003923:86 , srv878943204:86 , srv88820992:86 , srv915583672:86 , srv926454917:86 , srv98199212:86 , srv1045143294:87 , srv1357364179:87 , srv1375807515:87 , srv1424456934:87 , srv1592459843:87 , srv1645817243:87 , srv1695440458:87 , srv188467032:87 , srv1921035903:87 , srv2085775999:87 , srv2114469351:87 , srv528049723:87 , srv567497604:87 , srv790855607:87 , srv813337396:87 , srv833631637:87 , srv920990511:87 , srv958715051:87 , srv1121113790:88 , srv1521844820:88 , srv1875182482:88 , srv1704935445:557 } 2023-04-11 23:55:47,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-11 23:55:47,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1921035903=49, srv1121113790=6, srv60495680=77, srv1522936187=32, srv1427648031=24, srv472977346=71, srv426781502=66, srv1317815967=16, srv165499731=39, srv1875182482=46, srv1924962678=50, srv2085775999=57, srv528049723=74, srv1835122455=45, srv1494824752=29, srv980823327=98, srv148607108=27, srv1521844820=31, srv1578564626=35, srv1030992496=2, srv833631637=86, srv1357364179=20, srv678094308=79, srv813337396=85, srv1436576460=25, srv214943485=60, srv1544476851=33, srv389110227=64, srv1045143294=3, srv304983477=63, srv880851940=90, srv1915951514=48, srv2132781216=59, srv1424456934=23, srv1189181598=9, srv1375807515=21, srv1942560380=52, srv1247720753=10, srv1114427149=5, srv1948164000=53, srv1619237136=37, srv510818620=72, srv1514247213=30, srv2072927542=55, srv1316199073=15, srv919479903=93, srv958715051=97, srv188467032=47, srv1956422434=54, srv2114469351=58, srv1645817243=38, srv733434374=80, srv1704935445=41, srv462527659=70, srv2083539047=56, srv1155785667=8, srv1723221008=43, srv1019539513=1, srv1259716198=12, srv570971846=76, srv98199212=99, srv1339971715=18, srv1827386453=44, srv1270254316=13, srv1002011996=0, srv1285538288=14, srv1409314330=22, srv1439836670=26, srv279105715=62, srv920990511=94, srv526658146=73, srv44049629=68, srv1695440458=40, srv837003923=87, srv451924657=69, srv790855607=83, srv1133386197=7, srv88820992=91, srv926454917=95, srv62612927=78, srv1084987072=4, srv1934479171=51, srv567497604=75, srv942591694=96, srv1710362324=42, srv1592459843=36, srv802620233=84, srv2713158=61, srv764993443=81, srv1324583018=17, srv878943204=89, srv869923274=88, srv1256115793=11, srv1488510634=28, srv397352271=65, srv43116797=67, srv1354812701=19, srv915583672=92, srv1559523761=34, srv776360013=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-11 23:55:47,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-11 23:55:47,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-11 23:55:47,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-11 23:55:47,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-11 23:55:47,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-11 23:55:47,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-11 23:55:47,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-11 23:55:47,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-11 23:55:47,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-11 23:55:47,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-11 23:55:47,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-11 23:55:47,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-11 23:55:47,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-11 23:55:47,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-11 23:55:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-11 23:55:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-11 23:55:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-11 23:55:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-11 23:55:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-11 23:55:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-11 23:55:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-11 23:55:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-11 23:55:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-11 23:55:47,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-11 23:55:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-11 23:55:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-11 23:55:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-11 23:55:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-11 23:55:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-11 23:55:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-11 23:55:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-11 23:55:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-11 23:55:47,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-11 23:55:47,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-11 23:55:47,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-11 23:55:47,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-11 23:55:47,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-11 23:55:47,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-11 23:55:47,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-11 23:55:47,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-11 23:55:47,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-11 23:55:47,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-11 23:55:47,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-11 23:55:47,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-11 23:55:47,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-11 23:55:47,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-11 23:55:47,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-11 23:55:47,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-11 23:55:47,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-11 23:55:47,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-11 23:55:47,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-11 23:55:47,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-11 23:55:47,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-11 23:55:47,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-11 23:55:47,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-11 23:55:47,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-11 23:55:47,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-11 23:55:47,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-11 23:55:47,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-11 23:55:47,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-11 23:55:47,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-11 23:55:47,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-11 23:55:47,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-11 23:55:47,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-11 23:55:47,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-11 23:55:47,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-11 23:55:47,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-11 23:55:47,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-11 23:55:47,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-11 23:55:47,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-11 23:55:47,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-11 23:55:47,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-11 23:55:47,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-11 23:55:47,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-11 23:55:47,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-11 23:55:47,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-11 23:55:47,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-11 23:55:47,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-11 23:55:47,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-11 23:55:47,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-11 23:55:47,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-11 23:55:47,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-11 23:55:47,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-11 23:55:47,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-11 23:55:47,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-11 23:55:47,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-11 23:55:47,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-11 23:55:47,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-11 23:55:47,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-11 23:55:47,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-11 23:55:47,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-11 23:55:47,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-11 23:55:47,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-11 23:55:47,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-11 23:55:47,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-11 23:55:47,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-11 23:55:47,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-11 23:55:47,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-11 23:55:47,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-11 23:55:47,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-11 23:55:47,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-11 23:55:47,372 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005650148677010918); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0098016238147137); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005650148677010918); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:55:47,372 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-11 23:55:47,373 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.490120197360781E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005650148677010918); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0098016238147137); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005650148677010918); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-11 23:55:47,748 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 508 ms to try 5000 different iterations. Found a solution that moves 235 regions; Going from a computed imbalance of 6.490120197360781E-4 to a new imbalance of 2.7888529288435445E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.026111111111111113, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:55:47,749 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1019539513:83 , srv1030992496:83 , srv1439836670:83 , srv813337396:83 , srv1121113790:84 , srv1189181598:84 , srv1247720753:84 , srv1259716198:84 , srv1324583018:84 , srv1339971715:84 , srv1357364179:84 , srv1375807515:84 , srv1409314330:84 , srv1424456934:84 , srv148607108:84 , srv1514247213:84 , srv1521844820:84 , srv1559523761:84 , srv1619237136:84 , srv1645817243:84 , srv1695440458:84 , srv1723221008:84 , srv1875182482:84 , srv188467032:84 , srv1924962678:84 , srv1934479171:84 , srv1942560380:84 , srv1956422434:84 , srv2085775999:84 , srv2132781216:84 , srv279105715:84 , srv426781502:84 , srv44049629:84 , srv472977346:84 , srv567497604:84 , srv678094308:84 , srv733434374:84 , srv764993443:84 , srv776360013:84 , srv790855607:84 , srv802620233:84 , srv833631637:84 , srv880851940:84 , srv915583672:84 , srv919479903:84 , srv920990511:84 , srv942591694:84 , srv958715051:84 , srv98199212:84 , srv1002011996:85 , srv1045143294:85 , srv1084987072:85 , srv1114427149:85 , srv1133386197:85 , srv1256115793:85 , srv1270254316:85 , srv1285538288:85 , srv1316199073:85 , srv1317815967:85 , srv1354812701:85 , srv1427648031:85 , srv1436576460:85 , srv1488510634:85 , srv1494824752:85 , srv1522936187:85 , srv1544476851:85 , srv1578564626:85 , srv1592459843:85 , srv165499731:85 , srv1710362324:85 , srv1827386453:85 , srv1835122455:85 , srv1915951514:85 , srv1921035903:85 , srv1948164000:85 , srv2072927542:85 , srv2083539047:85 , srv2114469351:85 , srv214943485:85 , srv2713158:85 , srv304983477:85 , srv389110227:85 , srv397352271:85 , srv451924657:85 , srv462527659:85 , srv526658146:85 , srv528049723:85 , srv570971846:85 , srv60495680:85 , srv869923274:85 , srv878943204:85 , srv88820992:85 , srv926454917:85 , srv980823327:85 , srv1155785667:86 , srv43116797:86 , srv510818620:86 , srv62612927:86 , srv837003923:86 , srv1704935445:633 } 2023-04-11 23:55:47,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-11 23:55:47,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1921035903=49, srv1121113790=6, srv60495680=77, srv1522936187=32, srv1427648031=24, srv472977346=71, srv426781502=66, srv1317815967=16, srv165499731=39, srv1875182482=46, srv1924962678=50, srv2085775999=57, srv528049723=74, srv1835122455=45, srv1494824752=29, srv980823327=98, srv148607108=27, srv1521844820=31, srv1578564626=35, srv1030992496=2, srv833631637=86, srv1357364179=20, srv678094308=79, srv813337396=85, srv1436576460=25, srv214943485=60, srv1544476851=33, srv389110227=64, srv1045143294=3, srv304983477=63, srv880851940=90, srv1915951514=48, srv2132781216=59, srv1424456934=23, srv1189181598=9, srv1375807515=21, srv1942560380=52, srv1247720753=10, srv1114427149=5, srv1948164000=53, srv1619237136=37, srv510818620=72, srv1514247213=30, srv2072927542=55, srv1316199073=15, srv919479903=93, srv958715051=97, srv188467032=47, srv1956422434=54, srv2114469351=58, srv1645817243=38, srv733434374=80, srv1704935445=41, srv462527659=70, srv2083539047=56, srv1155785667=8, srv1723221008=43, srv1019539513=1, srv1259716198=12, srv570971846=76, srv98199212=99, srv1339971715=18, srv1827386453=44, srv1270254316=13, srv1002011996=0, srv1285538288=14, srv1409314330=22, srv1439836670=26, srv279105715=62, srv920990511=94, srv526658146=73, srv44049629=68, srv1695440458=40, srv837003923=87, srv451924657=69, srv790855607=83, srv1133386197=7, srv88820992=91, srv926454917=95, srv62612927=78, srv1084987072=4, srv1934479171=51, srv567497604=75, srv942591694=96, srv1710362324=42, srv1592459843=36, srv802620233=84, srv2713158=61, srv764993443=81, srv1324583018=17, srv878943204=89, srv869923274=88, srv1256115793=11, srv1488510634=28, srv397352271=65, srv43116797=67, srv1354812701=19, srv915583672=92, srv1559523761=34, srv776360013=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-11 23:55:47,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-11 23:55:47,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-11 23:55:47,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-11 23:55:47,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-11 23:55:47,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-11 23:55:47,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-11 23:55:47,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-11 23:55:47,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-11 23:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-11 23:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-11 23:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-11 23:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-11 23:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-11 23:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-11 23:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-11 23:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-11 23:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-11 23:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-11 23:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-11 23:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-11 23:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-11 23:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-11 23:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-11 23:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-11 23:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-11 23:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-11 23:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-11 23:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-11 23:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-11 23:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-11 23:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-11 23:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-11 23:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-11 23:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-11 23:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-11 23:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-11 23:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-11 23:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-11 23:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-11 23:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-11 23:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-11 23:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-11 23:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-11 23:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-11 23:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-11 23:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-11 23:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-11 23:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-11 23:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-11 23:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-11 23:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-11 23:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-11 23:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-11 23:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-11 23:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-11 23:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-11 23:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-11 23:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-11 23:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-11 23:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-11 23:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-11 23:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-11 23:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-11 23:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-11 23:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-11 23:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-11 23:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-11 23:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-11 23:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-11 23:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-11 23:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-11 23:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-11 23:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-11 23:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-11 23:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-11 23:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-11 23:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-11 23:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-11 23:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-11 23:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-11 23:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-11 23:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-11 23:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-11 23:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-11 23:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-11 23:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-11 23:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-11 23:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-11 23:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-11 23:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-11 23:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-11 23:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-11 23:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-11 23:55:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-11 23:55:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-11 23:55:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-11 23:55:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-11 23:55:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-11 23:55:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-11 23:55:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-11 23:55:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-11 23:55:47,889 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:55:47,889 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-11 23:55:47,889 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.624302411834823E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-11 23:55:48,153 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 364 ms to try 5000 different iterations. Found a solution that moves 25 regions; Going from a computed imbalance of 2.624302411834823E-5 to a new imbalance of 7.248293037828088E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.002777777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:55:48,154 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1488510634:83 , srv1710362324:83 , srv43116797:83 , srv510818620:83 , srv62612927:83 , srv1002011996:84 , srv1019539513:84 , srv1030992496:84 , srv1045143294:84 , srv1084987072:84 , srv1114427149:84 , srv1121113790:84 , srv1133386197:84 , srv1155785667:84 , srv1189181598:84 , srv1247720753:84 , srv1256115793:84 , srv1259716198:84 , srv1270254316:84 , srv1285538288:84 , srv1316199073:84 , srv1317815967:84 , srv1324583018:84 , srv1339971715:84 , srv1354812701:84 , srv1357364179:84 , srv1375807515:84 , srv1409314330:84 , srv1424456934:84 , srv1427648031:84 , srv1436576460:84 , srv148607108:84 , srv1494824752:84 , srv1514247213:84 , srv1521844820:84 , srv1522936187:84 , srv1544476851:84 , srv1559523761:84 , srv1578564626:84 , srv1592459843:84 , srv1619237136:84 , srv1645817243:84 , srv165499731:84 , srv1695440458:84 , srv1723221008:84 , srv1827386453:84 , srv1835122455:84 , srv1875182482:84 , srv188467032:84 , srv1915951514:84 , srv1921035903:84 , srv1924962678:84 , srv1934479171:84 , srv1942560380:84 , srv1948164000:84 , srv1956422434:84 , srv2072927542:84 , srv2083539047:84 , srv2085775999:84 , srv2114469351:84 , srv2132781216:84 , srv214943485:84 , srv2713158:84 , srv279105715:84 , srv304983477:84 , srv389110227:84 , srv397352271:84 , srv426781502:84 , srv44049629:84 , srv451924657:84 , srv462527659:84 , srv472977346:84 , srv526658146:84 , srv528049723:84 , srv567497604:84 , srv570971846:84 , srv60495680:84 , srv678094308:84 , srv733434374:84 , srv764993443:84 , srv790855607:84 , srv802620233:84 , srv813337396:84 , srv837003923:84 , srv869923274:84 , srv878943204:84 , srv880851940:84 , srv88820992:84 , srv915583672:84 , srv919479903:84 , srv920990511:84 , srv926454917:84 , srv958715051:84 , srv980823327:84 , srv98199212:84 , srv1439836670:85 , srv776360013:85 , srv833631637:85 , srv942591694:85 , srv1704935445:685 } 2023-04-11 23:55:48,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-11 23:55:48,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1921035903=49, srv1121113790=6, srv60495680=77, srv1522936187=32, srv1427648031=24, srv472977346=71, srv426781502=66, srv1317815967=16, srv165499731=39, srv1875182482=46, srv1924962678=50, srv2085775999=57, srv528049723=74, srv1835122455=45, srv1494824752=29, srv980823327=98, srv148607108=27, srv1521844820=31, srv1578564626=35, srv1030992496=2, srv833631637=86, srv1357364179=20, srv678094308=79, srv813337396=85, srv1436576460=25, srv214943485=60, srv1544476851=33, srv389110227=64, srv1045143294=3, srv304983477=63, srv880851940=90, srv1915951514=48, srv2132781216=59, srv1424456934=23, srv1189181598=9, srv1375807515=21, srv1942560380=52, srv1247720753=10, srv1114427149=5, srv1948164000=53, srv1619237136=37, srv510818620=72, srv1514247213=30, srv2072927542=55, srv1316199073=15, srv919479903=93, srv958715051=97, srv188467032=47, srv1956422434=54, srv2114469351=58, srv1645817243=38, srv733434374=80, srv1704935445=41, srv462527659=70, srv2083539047=56, srv1155785667=8, srv1723221008=43, srv1019539513=1, srv1259716198=12, srv570971846=76, srv98199212=99, srv1339971715=18, srv1827386453=44, srv1270254316=13, srv1002011996=0, srv1285538288=14, srv1409314330=22, srv1439836670=26, srv279105715=62, srv920990511=94, srv526658146=73, srv44049629=68, srv1695440458=40, srv837003923=87, srv451924657=69, srv790855607=83, srv1133386197=7, srv88820992=91, srv926454917=95, srv62612927=78, srv1084987072=4, srv1934479171=51, srv567497604=75, srv942591694=96, srv1710362324=42, srv1592459843=36, srv802620233=84, srv2713158=61, srv764993443=81, srv1324583018=17, srv878943204=89, srv869923274=88, srv1256115793=11, srv1488510634=28, srv397352271=65, srv43116797=67, srv1354812701=19, srv915583672=92, srv1559523761=34, srv776360013=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-11 23:55:48,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-11 23:55:48,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-11 23:55:48,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-11 23:55:48,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-11 23:55:48,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-11 23:55:48,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-11 23:55:48,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-11 23:55:48,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-11 23:55:48,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-11 23:55:48,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-11 23:55:48,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-11 23:55:48,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-11 23:55:48,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-11 23:55:48,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-11 23:55:48,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-11 23:55:48,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-11 23:55:48,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-11 23:55:48,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-11 23:55:48,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-11 23:55:48,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-11 23:55:48,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-11 23:55:48,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-11 23:55:48,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-11 23:55:48,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-11 23:55:48,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-11 23:55:48,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-11 23:55:48,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-11 23:55:48,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-11 23:55:48,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-11 23:55:48,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-11 23:55:48,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-11 23:55:48,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-11 23:55:48,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-11 23:55:48,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-11 23:55:48,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-11 23:55:48,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-11 23:55:48,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-11 23:55:48,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-11 23:55:48,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-11 23:55:48,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-11 23:55:48,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-11 23:55:48,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-11 23:55:48,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-11 23:55:48,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-11 23:55:48,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-11 23:55:48,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-11 23:55:48,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-11 23:55:48,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-11 23:55:48,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-11 23:55:48,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-11 23:55:48,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-11 23:55:48,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-11 23:55:48,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-11 23:55:48,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-11 23:55:48,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-11 23:55:48,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-11 23:55:48,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-11 23:55:48,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-11 23:55:48,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-11 23:55:48,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-11 23:55:48,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-11 23:55:48,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-11 23:55:48,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-11 23:55:48,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-11 23:55:48,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-11 23:55:48,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-11 23:55:48,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-11 23:55:48,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-11 23:55:48,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-11 23:55:48,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-11 23:55:48,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-11 23:55:48,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-11 23:55:48,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-11 23:55:48,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-11 23:55:48,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-11 23:55:48,319 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:55:48,319 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1488510634:83 , srv1710362324:83 , srv43116797:83 , srv510818620:83 , srv62612927:83 , srv1002011996:84 , srv1019539513:84 , srv1030992496:84 , srv1045143294:84 , srv1084987072:84 , srv1114427149:84 , srv1121113790:84 , srv1133386197:84 , srv1155785667:84 , srv1189181598:84 , srv1247720753:84 , srv1256115793:84 , srv1259716198:84 , srv1270254316:84 , srv1285538288:84 , srv1316199073:84 , srv1317815967:84 , srv1324583018:84 , srv1339971715:84 , srv1354812701:84 , srv1357364179:84 , srv1375807515:84 , srv1409314330:84 , srv1424456934:84 , srv1427648031:84 , srv1436576460:84 , srv148607108:84 , srv1494824752:84 , srv1514247213:84 , srv1521844820:84 , srv1522936187:84 , srv1544476851:84 , srv1559523761:84 , srv1578564626:84 , srv1592459843:84 , srv1619237136:84 , srv1645817243:84 , srv165499731:84 , srv1695440458:84 , srv1723221008:84 , srv1827386453:84 , srv1835122455:84 , srv1875182482:84 , srv188467032:84 , srv1915951514:84 , srv1921035903:84 , srv1924962678:84 , srv1934479171:84 , srv1942560380:84 , srv1948164000:84 , srv1956422434:84 , srv2072927542:84 , srv2083539047:84 , srv2085775999:84 , srv2114469351:84 , srv2132781216:84 , srv214943485:84 , srv2713158:84 , srv279105715:84 , srv304983477:84 , srv389110227:84 , srv397352271:84 , srv426781502:84 , srv44049629:84 , srv451924657:84 , srv462527659:84 , srv472977346:84 , srv526658146:84 , srv528049723:84 , srv567497604:84 , srv570971846:84 , srv60495680:84 , srv678094308:84 , srv733434374:84 , srv764993443:84 , srv790855607:84 , srv802620233:84 , srv813337396:84 , srv837003923:84 , srv869923274:84 , srv878943204:84 , srv880851940:84 , srv88820992:84 , srv915583672:84 , srv919479903:84 , srv920990511:84 , srv926454917:84 , srv958715051:84 , srv980823327:84 , srv98199212:84 , srv1439836670:85 , srv776360013:85 , srv833631637:85 , srv942591694:85 , srv1704935445:685 } 2023-04-11 23:55:48,348 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=361 (was 361), ProcessCount=169 (was 169), AvailableMemoryMB=1151 (was 1643) 2023-04-11 23:55:48,357 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=361, ProcessCount=169, AvailableMemoryMB=1151 2023-04-11 23:55:48,357 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-11 23:55:48,358 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-04-11 23:55:48,377 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1021419267:84 , srv1104211105:84 , srv1106532840:84 , srv123255536:84 , srv1261015373:84 , srv1316683340:84 , srv1375170715:84 , srv1456647891:84 , srv1529579023:84 , srv1603053568:84 , srv1618940487:84 , srv1791381757:84 , srv1839683491:84 , srv1901481072:84 , srv1938949249:84 , srv1996536320:84 , srv2084588789:84 , srv2109668873:84 , srv2113490025:84 , srv2117942626:84 , srv248273252:84 , srv311560529:84 , srv441772851:84 , srv445223881:84 , srv708708247:84 , srv716253028:84 , srv720626079:84 , srv906321079:84 , srv915107433:84 , srv372957488:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-04-11 23:55:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-11 23:55:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1901481072=13, srv248273252=20, srv445223881=24, srv1375170715=6, srv1456647891=7, srv1316683340=5, srv1106532840=2, srv716253028=26, srv1938949249=14, srv2109668873=17, srv1104211105=1, srv1261015373=4, srv708708247=25, srv2084588789=16, srv2113490025=18, srv2117942626=19, srv1839683491=12, srv311560529=21, srv123255536=3, srv1603053568=9, srv906321079=28, srv1996536320=15, srv1791381757=11, srv720626079=27, srv1618940487=10, srv1529579023=8, srv441772851=23, srv372957488=22, srv1021419267=0, srv915107433=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-11 23:55:48,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-11 23:55:48,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-11 23:55:48,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-11 23:55:48,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-11 23:55:48,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-11 23:55:48,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-11 23:55:48,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-11 23:55:48,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-11 23:55:48,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-11 23:55:48,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-11 23:55:48,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-11 23:55:48,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-11 23:55:48,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-11 23:55:48,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-11 23:55:48,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-11 23:55:48,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-11 23:55:48,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-11 23:55:48,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-11 23:55:48,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-11 23:55:48,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-11 23:55:48,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-11 23:55:48,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-11 23:55:48,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-11 23:55:48,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-11 23:55:48,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-11 23:55:48,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-11 23:55:48,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-11 23:55:48,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-11 23:55:48,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-11 23:55:48,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-11 23:55:48,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-11 23:55:48,435 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1689378306216123, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:55:48,435 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991272536085224, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1689378306216123, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-04-11 23:57:48,405 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6106591 different iterations. Found a solution that moves 1859 regions; Going from a computed imbalance of 0.991272536085224 to a new imbalance of 4.338998739279625E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6885185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:57:48,414 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1021419267:90 , srv1104211105:90 , srv1106532840:90 , srv123255536:90 , srv1261015373:90 , srv1316683340:90 , srv1375170715:90 , srv1456647891:90 , srv1529579023:90 , srv1603053568:90 , srv1618940487:90 , srv1791381757:90 , srv1839683491:90 , srv1901481072:90 , srv1938949249:90 , srv1996536320:90 , srv2084588789:90 , srv2109668873:90 , srv2113490025:90 , srv2117942626:90 , srv248273252:90 , srv311560529:90 , srv372957488:90 , srv441772851:90 , srv445223881:90 , srv708708247:90 , srv716253028:90 , srv720626079:90 , srv906321079:90 , srv915107433:90 } 2023-04-11 23:57:48,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-11 23:57:48,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1901481072=13, srv248273252=20, srv445223881=24, srv1375170715=6, srv1456647891=7, srv1316683340=5, srv1106532840=2, srv716253028=26, srv1938949249=14, srv2109668873=17, srv1104211105=1, srv1261015373=4, srv708708247=25, srv2084588789=16, srv2113490025=18, srv2117942626=19, srv1839683491=12, srv311560529=21, srv123255536=3, srv1603053568=9, srv906321079=28, srv1996536320=15, srv1791381757=11, srv720626079=27, srv1618940487=10, srv1529579023=8, srv441772851=23, srv372957488=22, srv1021419267=0, srv915107433=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-11 23:57:48,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-11 23:57:48,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-11 23:57:48,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-11 23:57:48,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-11 23:57:48,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-11 23:57:48,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-11 23:57:48,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-11 23:57:48,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-11 23:57:48,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-11 23:57:48,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-11 23:57:48,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-11 23:57:48,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-11 23:57:48,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-11 23:57:48,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-11 23:57:48,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-11 23:57:48,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-11 23:57:48,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-11 23:57:48,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-11 23:57:48,441 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-11 23:57:48,441 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1021419267:90 , srv1104211105:90 , srv1106532840:90 , srv123255536:90 , srv1261015373:90 , srv1316683340:90 , srv1375170715:90 , srv1456647891:90 , srv1529579023:90 , srv1603053568:90 , srv1618940487:90 , srv1791381757:90 , srv1839683491:90 , srv1901481072:90 , srv1938949249:90 , srv1996536320:90 , srv2084588789:90 , srv2109668873:90 , srv2113490025:90 , srv2117942626:90 , srv248273252:90 , srv311560529:90 , srv372957488:90 , srv441772851:90 , srv445223881:90 , srv708708247:90 , srv716253028:90 , srv720626079:90 , srv906321079:90 , srv915107433:90 } 2023-04-11 23:57:48,455 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=286 (was 361), ProcessCount=169 (was 169), AvailableMemoryMB=1211 (was 1151) - AvailableMemoryMB LEAK? -