2024-01-29 17:53:12,311 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-01-29 17:53:12,630 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-29 17:53:12,752 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-29 17:53:12,817 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-29 17:53:12,825 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-29 17:53:12,839 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=325, ProcessCount=167, AvailableMemoryMB=3412 2024-01-29 17:53:12,841 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-29 17:53:12,842 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-29 17:53:12,976 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv102049386:84 , srv1026943357:84 , srv1032341052:84 , srv104223132:84 , srv1064663970:84 , srv1080640518:84 , srv1085790042:84 , srv1101052629:84 , srv1111339670:84 , srv1127831181:84 , srv1133957545:84 , srv1140188385:84 , srv1141342355:84 , srv1150311305:84 , srv1172915334:84 , srv1222948748:84 , srv1229616715:84 , srv1278629659:84 , srv1287352322:84 , srv129254065:84 , srv1378014803:84 , srv1395377021:84 , srv1452915897:84 , srv1462507729:84 , srv1489980787:84 , srv1490922483:84 , srv1503015092:84 , srv1554038369:84 , srv1593747683:84 , srv1594737754:84 , srv1607537184:84 , srv1607856678:84 , srv1609235040:84 , srv1621081396:84 , srv1641110001:84 , srv1647920268:84 , srv1661193335:84 , srv166272086:84 , srv1666211746:84 , srv1690732990:84 , srv1755455220:84 , srv1760730239:84 , srv1836687867:84 , srv1875292925:84 , srv1914951070:84 , srv1915338067:84 , srv1919616772:84 , srv1935807631:84 , srv1942024522:84 , srv1951442903:84 , srv1965988692:84 , srv1976190577:84 , srv1996083973:84 , srv1999594557:84 , srv202798712:84 , srv2044684766:84 , srv2062328510:84 , srv2078597288:84 , srv2080674738:84 , srv2084322036:84 , srv2091259417:84 , srv2127039386:84 , srv2143522717:84 , srv232142392:84 , srv244207362:84 , srv283235678:84 , srv294066485:84 , srv317535327:84 , srv390184672:84 , srv448183756:84 , srv448910972:84 , srv475703425:84 , srv48728489:84 , srv500573581:84 , srv52124561:84 , srv538727160:84 , srv540844392:84 , srv542462581:84 , srv552239242:84 , srv568164478:84 , srv584867229:84 , srv590812418:84 , srv59161497:84 , srv597304922:84 , srv658931274:84 , srv684699676:84 , srv732229731:84 , srv741461827:84 , srv747395394:84 , srv757694362:84 , srv843974300:84 , srv878468268:84 , srv889324366:84 , srv889833795:84 , srv893072344:84 , srv937564302:84 , srv996344196:84 , srv997826431:84 , srv999161768:84 , srv1353378411:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-01-29 17:53:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 17:53:12,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1127831181=9, srv2127039386=62, srv1378014803=21, srv1593747683=29, srv2084322036=60, srv878468268=92, srv500573581=74, srv590812418=82, srv448183756=70, srv1666211746=39, srv1951442903=50, srv2062328510=57, srv1222948748=15, srv1101052629=7, srv48728489=73, srv597304922=84, srv889324366=93, srv244207362=65, srv1621081396=34, srv104223132=3, srv1111339670=8, srv1609235040=33, srv1287352322=18, srv584867229=81, srv1641110001=35, srv889833795=94, srv542462581=78, srv1080640518=5, srv129254065=19, srv1935807631=48, srv1278629659=17, srv1026943357=1, srv1133957545=10, srv475703425=72, srv1353378411=20, srv1032341052=2, srv166272086=38, srv1690732990=40, srv1490922483=26, srv1914951070=45, srv843974300=91, srv658931274=85, srv2143522717=63, srv59161497=83, srv1875292925=44, srv390184672=69, srv684699676=86, srv732229731=87, srv1647920268=36, srv568164478=80, srv2044684766=56, srv1965988692=51, srv1607856678=32, srv552239242=79, srv1172915334=14, srv540844392=77, srv52124561=75, srv999161768=99, srv1661193335=37, srv294066485=67, srv1140188385=11, srv1836687867=43, srv102049386=0, srv1150311305=13, srv2080674738=59, srv1462507729=24, srv538727160=76, srv757694362=90, srv747395394=89, srv1064663970=4, srv1452915897=23, srv2078597288=58, srv1755455220=41, srv1085790042=6, srv1395377021=22, srv232142392=64, srv893072344=95, srv1976190577=52, srv1229616715=16, srv1942024522=49, srv1141342355=12, srv1999594557=54, srv741461827=88, srv2091259417=61, srv317535327=68, srv1489980787=25, srv448910972=71, srv996344196=97, srv1594737754=30, srv1760730239=42, srv1919616772=47, srv997826431=98, srv283235678=66, srv1996083973=53, srv1554038369=28, srv1503015092=27, srv937564302=96, srv202798712=55, srv1915338067=46, srv1607537184=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 17:53:13,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 17:53:13,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 17:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 17:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 17:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 17:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 17:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 17:53:13,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 17:53:13,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 17:53:13,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 17:53:13,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 17:53:13,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 17:53:13,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 17:53:13,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 17:53:13,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 17:53:13,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 17:53:13,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 17:53:13,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 17:53:13,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 17:53:13,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 17:53:13,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 17:53:13,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 17:53:13,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 17:53:13,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 17:53:13,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 17:53:13,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 17:53:13,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 17:53:13,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 17:53:13,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 17:53:13,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 17:53:13,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-29 17:53:13,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-29 17:53:13,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-29 17:53:13,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-29 17:53:13,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-29 17:53:13,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-29 17:53:13,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-29 17:53:13,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-29 17:53:13,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-29 17:53:13,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-29 17:53:13,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-29 17:53:13,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-29 17:53:13,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-29 17:53:13,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-29 17:53:13,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-29 17:53:13,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-29 17:53:13,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-29 17:53:13,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-29 17:53:13,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-29 17:53:13,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-29 17:53:13,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-29 17:53:13,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-29 17:53:13,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-29 17:53:13,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-29 17:53:13,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-29 17:53:13,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-29 17:53:13,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-29 17:53:13,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-29 17:53:13,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-29 17:53:13,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-29 17:53:13,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-29 17:53:13,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-29 17:53:13,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-29 17:53:13,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-29 17:53:13,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-29 17:53:13,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-29 17:53:13,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-29 17:53:13,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-29 17:53:13,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-29 17:53:13,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-29 17:53:13,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-29 17:53:13,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-29 17:53:13,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-29 17:53:13,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-29 17:53:13,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-29 17:53:13,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-29 17:53:13,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-29 17:53:13,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-29 17:53:13,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-29 17:53:13,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-29 17:53:13,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-29 17:53:13,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-29 17:53:13,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-29 17:53:13,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-29 17:53:13,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-29 17:53:13,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-29 17:53:13,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-29 17:53:13,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-29 17:53:13,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-29 17:53:13,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-29 17:53:13,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-29 17:53:13,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-29 17:53:13,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-29 17:53:13,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-29 17:53:13,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-29 17:53:13,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-29 17:53:13,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-29 17:53:13,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-29 17:53:13,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-29 17:53:13,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-29 17:53:13,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-29 17:53:13,525 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:53:13,525 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-29 17:53:13,526 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-29 17:53:13,893 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 755 ms to try 5000 different iterations. Found a solution that moves 4682 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18177507814904142. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03985070029502919, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05880401736201752, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5202222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03985070029502919, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17691666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:53:13,916 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv475703425:81 , srv684699676:81 , srv1287352322:82 , srv1593747683:82 , srv997826431:82 , srv1760730239:83 , srv540844392:83 , srv568164478:83 , srv59161497:83 , srv658931274:83 , srv102049386:84 , srv1133957545:84 , srv1150311305:84 , srv1278629659:84 , srv1378014803:84 , srv1395377021:84 , srv1489980787:84 , srv1594737754:84 , srv1915338067:84 , srv2078597288:84 , srv244207362:84 , srv283235678:84 , srv48728489:84 , srv552239242:84 , srv889833795:84 , srv104223132:85 , srv1080640518:85 , srv1101052629:85 , srv1172915334:85 , srv129254065:85 , srv1609235040:85 , srv1621081396:85 , srv1976190577:85 , srv202798712:85 , srv2080674738:85 , srv732229731:85 , srv741461827:85 , srv747395394:85 , srv996344196:85 , srv999161768:85 , srv1026943357:86 , srv1140188385:86 , srv1141342355:86 , srv1222948748:86 , srv1490922483:86 , srv1942024522:86 , srv1951442903:86 , srv1996083973:86 , srv2084322036:86 , srv2091259417:86 , srv294066485:86 , srv500573581:86 , srv538727160:86 , srv542462581:86 , srv889324366:86 , srv1032341052:87 , srv1064663970:87 , srv1607537184:87 , srv1607856678:87 , srv1661193335:87 , srv166272086:87 , srv1935807631:87 , srv2062328510:87 , srv2127039386:87 , srv2143522717:87 , srv390184672:87 , srv448183756:87 , srv448910972:87 , srv52124561:87 , srv590812418:87 , srv843974300:87 , srv1085790042:88 , srv1127831181:88 , srv1229616715:88 , srv1452915897:88 , srv1647920268:88 , srv1690732990:88 , srv1755455220:88 , srv1914951070:88 , srv1919616772:88 , srv1965988692:88 , srv1999594557:88 , srv584867229:88 , srv1111339670:89 , srv1462507729:89 , srv1641110001:89 , srv757694362:89 , srv1666211746:90 , srv1554038369:91 , srv1875292925:91 , srv317535327:91 , srv878468268:91 , srv893072344:91 , srv1503015092:92 , srv2044684766:92 , srv597304922:92 , srv937564302:92 , srv232142392:95 , srv1836687867:96 , srv1353378411:444 } 2024-01-29 17:53:13,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 17:53:13,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1127831181=9, srv2127039386=62, srv1378014803=21, srv1593747683=29, srv2084322036=60, srv878468268=92, srv500573581=74, srv590812418=82, srv448183756=70, srv1666211746=39, srv1951442903=50, srv2062328510=57, srv1222948748=15, srv1101052629=7, srv48728489=73, srv597304922=84, srv889324366=93, srv244207362=65, srv1621081396=34, srv104223132=3, srv1111339670=8, srv1609235040=33, srv1287352322=18, srv584867229=81, srv1641110001=35, srv889833795=94, srv542462581=78, srv1080640518=5, srv129254065=19, srv1935807631=48, srv1278629659=17, srv1026943357=1, srv1133957545=10, srv475703425=72, srv1353378411=20, srv1032341052=2, srv166272086=38, srv1690732990=40, srv1490922483=26, srv1914951070=45, srv843974300=91, srv658931274=85, srv2143522717=63, srv59161497=83, srv1875292925=44, srv390184672=69, srv684699676=86, srv732229731=87, srv1647920268=36, srv568164478=80, srv2044684766=56, srv1965988692=51, srv1607856678=32, srv552239242=79, srv1172915334=14, srv540844392=77, srv52124561=75, srv999161768=99, srv1661193335=37, srv294066485=67, srv1140188385=11, srv1836687867=43, srv102049386=0, srv1150311305=13, srv2080674738=59, srv1462507729=24, srv538727160=76, srv757694362=90, srv747395394=89, srv1064663970=4, srv1452915897=23, srv2078597288=58, srv1755455220=41, srv1085790042=6, srv1395377021=22, srv232142392=64, srv893072344=95, srv1976190577=52, srv1229616715=16, srv1942024522=49, srv1141342355=12, srv1999594557=54, srv741461827=88, srv2091259417=61, srv317535327=68, srv1489980787=25, srv448910972=71, srv996344196=97, srv1594737754=30, srv1760730239=42, srv1919616772=47, srv997826431=98, srv283235678=66, srv1996083973=53, srv1554038369=28, srv1503015092=27, srv937564302=96, srv202798712=55, srv1915338067=46, srv1607537184=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 17:53:13,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 17:53:13,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 17:53:13,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 17:53:13,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 17:53:13,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 17:53:13,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 17:53:13,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 17:53:13,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 17:53:13,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 17:53:13,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 17:53:13,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 17:53:13,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 17:53:13,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 17:53:13,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 17:53:13,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 17:53:13,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 17:53:13,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 17:53:13,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 17:53:13,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 17:53:13,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 17:53:13,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 17:53:13,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 17:53:13,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 17:53:13,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 17:53:13,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 17:53:13,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 17:53:13,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 17:53:13,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 17:53:13,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 17:53:13,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 17:53:13,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-29 17:53:13,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-29 17:53:13,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-29 17:53:13,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-29 17:53:13,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-29 17:53:13,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-29 17:53:13,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-29 17:53:13,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-29 17:53:13,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-29 17:53:13,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-29 17:53:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-29 17:53:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-29 17:53:13,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-29 17:53:13,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-29 17:53:13,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-29 17:53:13,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-29 17:53:13,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-29 17:53:13,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-29 17:53:13,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-29 17:53:13,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-29 17:53:13,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-29 17:53:13,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-29 17:53:13,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-29 17:53:13,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-29 17:53:13,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-29 17:53:13,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-29 17:53:13,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-29 17:53:13,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-29 17:53:13,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-29 17:53:13,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-29 17:53:14,065 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03985070029502919, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05880401736201752, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03985070029502919, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17691666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:53:14,066 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-29 17:53:14,066 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18174229408433354, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03985070029502919, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05880401736201752, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03985070029502919, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17691666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-29 17:53:14,320 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 354 ms to try 5000 different iterations. Found a solution that moves 2605 regions; Going from a computed imbalance of 0.18174229408433354 to a new imbalance of 0.016189785670681193. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02148670550667006); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034582648770150555, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28944444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02148670550667006); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:53:14,332 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1836687867:74 , srv232142392:76 , srv317535327:80 , srv1875292925:81 , srv2044684766:81 , srv757694362:81 , srv1229616715:82 , srv1503015092:82 , srv1647920268:82 , srv1666211746:82 , srv1755455220:82 , srv878468268:82 , srv937564302:82 , srv1111339670:83 , srv1462507729:83 , srv1554038369:83 , srv1641110001:83 , srv584867229:83 , srv597304922:83 , srv893072344:83 , srv1064663970:84 , srv1085790042:84 , srv1919616772:84 , srv1999594557:84 , srv500573581:84 , srv538727160:84 , srv843974300:84 , srv889324366:84 , srv1026943357:85 , srv1127831181:85 , srv1140188385:85 , srv1172915334:85 , srv1661193335:85 , srv166272086:85 , srv1914951070:85 , srv1942024522:85 , srv1951442903:85 , srv1965988692:85 , srv1996083973:85 , srv2091259417:85 , srv2127039386:85 , srv294066485:85 , srv590812418:85 , srv741461827:85 , srv996344196:85 , srv1032341052:86 , srv1141342355:86 , srv1489980787:86 , srv1490922483:86 , srv1607856678:86 , srv1935807631:86 , srv202798712:86 , srv2062328510:86 , srv2080674738:86 , srv390184672:86 , srv747395394:86 , srv102049386:87 , srv1080640518:87 , srv1133957545:87 , srv1222948748:87 , srv129254065:87 , srv1378014803:87 , srv1452915897:87 , srv1594737754:87 , srv1607537184:87 , srv1609235040:87 , srv1621081396:87 , srv1690732990:87 , srv1976190577:87 , srv2143522717:87 , srv448183756:87 , srv52124561:87 , srv542462581:87 , srv732229731:87 , srv999161768:87 , srv104223132:88 , srv1101052629:88 , srv1150311305:88 , srv1278629659:88 , srv1760730239:88 , srv2078597288:88 , srv2084322036:88 , srv244207362:88 , srv283235678:88 , srv448910972:88 , srv48728489:88 , srv552239242:88 , srv59161497:88 , srv889833795:88 , srv1395377021:89 , srv1593747683:89 , srv1915338067:89 , srv568164478:89 , srv658931274:89 , srv540844392:90 , srv475703425:91 , srv997826431:91 , srv1287352322:93 , srv684699676:93 , srv1353378411:521 } 2024-01-29 17:53:14,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 17:53:14,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1127831181=9, srv2127039386=62, srv1378014803=21, srv1593747683=29, srv2084322036=60, srv878468268=92, srv500573581=74, srv590812418=82, srv448183756=70, srv1666211746=39, srv1951442903=50, srv2062328510=57, srv1222948748=15, srv1101052629=7, srv48728489=73, srv597304922=84, srv889324366=93, srv244207362=65, srv1621081396=34, srv104223132=3, srv1111339670=8, srv1609235040=33, srv1287352322=18, srv584867229=81, srv1641110001=35, srv889833795=94, srv542462581=78, srv1080640518=5, srv129254065=19, srv1935807631=48, srv1278629659=17, srv1026943357=1, srv1133957545=10, srv475703425=72, srv1353378411=20, srv1032341052=2, srv166272086=38, srv1690732990=40, srv1490922483=26, srv1914951070=45, srv843974300=91, srv658931274=85, srv2143522717=63, srv59161497=83, srv1875292925=44, srv390184672=69, srv684699676=86, srv732229731=87, srv1647920268=36, srv568164478=80, srv2044684766=56, srv1965988692=51, srv1607856678=32, srv552239242=79, srv1172915334=14, srv540844392=77, srv52124561=75, srv999161768=99, srv1661193335=37, srv294066485=67, srv1140188385=11, srv1836687867=43, srv102049386=0, srv1150311305=13, srv2080674738=59, srv1462507729=24, srv538727160=76, srv757694362=90, srv747395394=89, srv1064663970=4, srv1452915897=23, srv2078597288=58, srv1755455220=41, srv1085790042=6, srv1395377021=22, srv232142392=64, srv893072344=95, srv1976190577=52, srv1229616715=16, srv1942024522=49, srv1141342355=12, srv1999594557=54, srv741461827=88, srv2091259417=61, srv317535327=68, srv1489980787=25, srv448910972=71, srv996344196=97, srv1594737754=30, srv1760730239=42, srv1919616772=47, srv997826431=98, srv283235678=66, srv1996083973=53, srv1554038369=28, srv1503015092=27, srv937564302=96, srv202798712=55, srv1915338067=46, srv1607537184=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 17:53:14,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 17:53:14,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 17:53:14,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 17:53:14,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 17:53:14,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 17:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 17:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 17:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 17:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 17:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 17:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 17:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 17:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 17:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 17:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 17:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 17:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 17:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 17:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 17:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 17:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 17:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 17:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 17:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 17:53:14,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 17:53:14,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 17:53:14,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 17:53:14,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 17:53:14,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 17:53:14,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 17:53:14,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-29 17:53:14,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-29 17:53:14,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-29 17:53:14,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-29 17:53:14,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-29 17:53:14,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-29 17:53:14,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-29 17:53:14,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-29 17:53:14,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-29 17:53:14,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-29 17:53:14,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-29 17:53:14,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-29 17:53:14,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-29 17:53:14,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-29 17:53:14,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-29 17:53:14,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-29 17:53:14,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-29 17:53:14,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-29 17:53:14,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-29 17:53:14,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-29 17:53:14,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-29 17:53:14,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-29 17:53:14,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-29 17:53:14,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-29 17:53:14,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-29 17:53:14,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-29 17:53:14,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-29 17:53:14,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-29 17:53:14,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-29 17:53:14,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-29 17:53:14,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-29 17:53:14,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-29 17:53:14,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-29 17:53:14,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-29 17:53:14,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-29 17:53:14,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-29 17:53:14,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-29 17:53:14,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-29 17:53:14,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-29 17:53:14,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-29 17:53:14,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-29 17:53:14,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-29 17:53:14,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-29 17:53:14,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-29 17:53:14,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-29 17:53:14,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-29 17:53:14,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-29 17:53:14,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-29 17:53:14,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-29 17:53:14,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-29 17:53:14,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-29 17:53:14,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-29 17:53:14,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-29 17:53:14,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-29 17:53:14,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-29 17:53:14,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-29 17:53:14,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-29 17:53:14,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-29 17:53:14,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-29 17:53:14,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-29 17:53:14,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-29 17:53:14,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-29 17:53:14,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-29 17:53:14,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-29 17:53:14,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-29 17:53:14,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-29 17:53:14,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-29 17:53:14,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-29 17:53:14,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-29 17:53:14,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-29 17:53:14,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-29 17:53:14,530 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02148670550667006); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034582648770150555, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02148670550667006); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:53:14,531 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-29 17:53:14,531 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016171545070817035, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02148670550667006); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034582648770150555, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02148670550667006); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-29 17:53:14,767 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 369 ms to try 5000 different iterations. Found a solution that moves 1010 regions; Going from a computed imbalance of 0.016171545070817035 to a new imbalance of 6.916511346949085E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006655433858912234); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009960640836865495); MoveCostFunction : (multiplier=7.0, imbalance=0.11222222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006655433858912234); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:53:14,770 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv597304922:81 , srv1690732990:82 , srv1287352322:83 , srv1452915897:83 , srv1503015092:83 , srv1554038369:83 , srv1607537184:83 , srv2084322036:83 , srv448910972:83 , srv52124561:83 , srv684699676:83 , srv937564302:83 , srv1032341052:84 , srv1127831181:84 , srv1395377021:84 , srv1935807631:84 , srv1965988692:84 , srv2044684766:84 , srv2143522717:84 , srv390184672:84 , srv448183756:84 , srv540844392:84 , srv542462581:84 , srv568164478:84 , srv893072344:84 , srv104223132:85 , srv1080640518:85 , srv1085790042:85 , srv1101052629:85 , srv1111339670:85 , srv1141342355:85 , srv1150311305:85 , srv1222948748:85 , srv129254065:85 , srv1462507729:85 , srv1490922483:85 , srv1607856678:85 , srv1609235040:85 , srv1621081396:85 , srv1641110001:85 , srv1914951070:85 , srv1915338067:85 , srv1976190577:85 , srv1999594557:85 , srv2078597288:85 , srv2127039386:85 , srv475703425:85 , srv48728489:85 , srv590812418:85 , srv732229731:85 , srv878468268:85 , srv997826431:85 , srv102049386:86 , srv1026943357:86 , srv1064663970:86 , srv1133957545:86 , srv1140188385:86 , srv1278629659:86 , srv1593747683:86 , srv1594737754:86 , srv1661193335:86 , srv166272086:86 , srv1666211746:86 , srv1760730239:86 , srv1875292925:86 , srv1942024522:86 , srv1951442903:86 , srv1996083973:86 , srv202798712:86 , srv2062328510:86 , srv2080674738:86 , srv232142392:86 , srv244207362:86 , srv283235678:86 , srv294066485:86 , srv317535327:86 , srv552239242:86 , srv584867229:86 , srv658931274:86 , srv889833795:86 , srv999161768:86 , srv1229616715:87 , srv1378014803:87 , srv1755455220:87 , srv1836687867:87 , srv1919616772:87 , srv2091259417:87 , srv59161497:87 , srv741461827:87 , srv747395394:87 , srv889324366:87 , srv996344196:87 , srv1172915334:88 , srv1489980787:88 , srv500573581:88 , srv538727160:88 , srv757694362:88 , srv843974300:88 , srv1647920268:89 , srv1353378411:552 } 2024-01-29 17:53:14,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 17:53:14,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1127831181=9, srv2127039386=62, srv1378014803=21, srv1593747683=29, srv2084322036=60, srv878468268=92, srv500573581=74, srv590812418=82, srv448183756=70, srv1666211746=39, srv1951442903=50, srv2062328510=57, srv1222948748=15, srv1101052629=7, srv48728489=73, srv597304922=84, srv889324366=93, srv244207362=65, srv1621081396=34, srv104223132=3, srv1111339670=8, srv1609235040=33, srv1287352322=18, srv584867229=81, srv1641110001=35, srv889833795=94, srv542462581=78, srv1080640518=5, srv129254065=19, srv1935807631=48, srv1278629659=17, srv1026943357=1, srv1133957545=10, srv475703425=72, srv1353378411=20, srv1032341052=2, srv166272086=38, srv1690732990=40, srv1490922483=26, srv1914951070=45, srv843974300=91, srv658931274=85, srv2143522717=63, srv59161497=83, srv1875292925=44, srv390184672=69, srv684699676=86, srv732229731=87, srv1647920268=36, srv568164478=80, srv2044684766=56, srv1965988692=51, srv1607856678=32, srv552239242=79, srv1172915334=14, srv540844392=77, srv52124561=75, srv999161768=99, srv1661193335=37, srv294066485=67, srv1140188385=11, srv1836687867=43, srv102049386=0, srv1150311305=13, srv2080674738=59, srv1462507729=24, srv538727160=76, srv757694362=90, srv747395394=89, srv1064663970=4, srv1452915897=23, srv2078597288=58, srv1755455220=41, srv1085790042=6, srv1395377021=22, srv232142392=64, srv893072344=95, srv1976190577=52, srv1229616715=16, srv1942024522=49, srv1141342355=12, srv1999594557=54, srv741461827=88, srv2091259417=61, srv317535327=68, srv1489980787=25, srv448910972=71, srv996344196=97, srv1594737754=30, srv1760730239=42, srv1919616772=47, srv997826431=98, srv283235678=66, srv1996083973=53, srv1554038369=28, srv1503015092=27, srv937564302=96, srv202798712=55, srv1915338067=46, srv1607537184=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 17:53:14,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 17:53:14,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 17:53:14,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 17:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 17:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 17:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 17:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 17:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 17:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 17:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 17:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 17:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 17:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 17:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 17:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 17:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 17:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 17:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 17:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 17:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 17:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 17:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 17:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 17:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 17:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 17:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 17:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 17:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 17:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 17:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 17:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-29 17:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-29 17:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-29 17:53:14,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-29 17:53:14,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-29 17:53:14,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-29 17:53:14,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-29 17:53:14,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-29 17:53:14,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-29 17:53:14,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-29 17:53:14,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-29 17:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-29 17:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-29 17:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-29 17:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-29 17:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-29 17:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-29 17:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-29 17:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-29 17:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-29 17:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-29 17:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-29 17:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-29 17:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-29 17:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-29 17:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-29 17:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-29 17:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-29 17:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-29 17:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-29 17:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-29 17:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-29 17:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-29 17:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-29 17:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-29 17:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-29 17:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-29 17:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-29 17:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-29 17:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-29 17:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-29 17:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-29 17:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-29 17:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-29 17:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-29 17:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-29 17:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-29 17:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-29 17:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-29 17:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-29 17:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-29 17:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-29 17:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-29 17:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-29 17:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-29 17:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-29 17:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-29 17:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-29 17:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-29 17:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-29 17:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-29 17:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-29 17:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-29 17:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-29 17:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-29 17:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-29 17:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-29 17:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-29 17:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-29 17:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-29 17:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-29 17:53:14,954 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006655433858912234); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009960640836865495); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006655433858912234); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:53:14,954 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-29 17:53:14,956 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.845789635383634E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006655433858912234); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009960640836865495); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006655433858912234); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-29 17:53:15,392 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 572 ms to try 5000 different iterations. Found a solution that moves 248 regions; Going from a computed imbalance of 6.845789635383634E-4 to a new imbalance of 5.702260551185655E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.027555555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:53:15,393 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1647920268:83 , srv1875292925:83 , srv2062328510:83 , srv843974300:83 , srv1026943357:84 , srv104223132:84 , srv1101052629:84 , srv1172915334:84 , srv1222948748:84 , srv1278629659:84 , srv1287352322:84 , srv1378014803:84 , srv1452915897:84 , srv1462507729:84 , srv1489980787:84 , srv1607856678:84 , srv1661193335:84 , srv166272086:84 , srv1666211746:84 , srv1690732990:84 , srv1914951070:84 , srv1919616772:84 , srv1999594557:84 , srv2091259417:84 , srv2143522717:84 , srv244207362:84 , srv283235678:84 , srv448183756:84 , srv448910972:84 , srv500573581:84 , srv538727160:84 , srv552239242:84 , srv59161497:84 , srv658931274:84 , srv747395394:84 , srv757694362:84 , srv878468268:84 , srv889324366:84 , srv889833795:84 , srv893072344:84 , srv937564302:84 , srv997826431:84 , srv999161768:84 , srv102049386:85 , srv1032341052:85 , srv1064663970:85 , srv1080640518:85 , srv1085790042:85 , srv1111339670:85 , srv1127831181:85 , srv1140188385:85 , srv1141342355:85 , srv1229616715:85 , srv129254065:85 , srv1395377021:85 , srv1490922483:85 , srv1554038369:85 , srv1593747683:85 , srv1594737754:85 , srv1607537184:85 , srv1609235040:85 , srv1621081396:85 , srv1641110001:85 , srv1755455220:85 , srv1760730239:85 , srv1836687867:85 , srv1915338067:85 , srv1935807631:85 , srv1942024522:85 , srv1951442903:85 , srv1965988692:85 , srv1976190577:85 , srv1996083973:85 , srv2044684766:85 , srv2078597288:85 , srv2080674738:85 , srv2084322036:85 , srv2127039386:85 , srv232142392:85 , srv294066485:85 , srv317535327:85 , srv390184672:85 , srv475703425:85 , srv48728489:85 , srv52124561:85 , srv540844392:85 , srv542462581:85 , srv584867229:85 , srv590812418:85 , srv684699676:85 , srv732229731:85 , srv741461827:85 , srv996344196:85 , srv1133957545:86 , srv1150311305:86 , srv1503015092:86 , srv202798712:86 , srv568164478:86 , srv597304922:86 , srv1353378411:626 } 2024-01-29 17:53:15,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 17:53:15,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1127831181=9, srv2127039386=62, srv1378014803=21, srv1593747683=29, srv2084322036=60, srv878468268=92, srv500573581=74, srv590812418=82, srv448183756=70, srv1666211746=39, srv1951442903=50, srv2062328510=57, srv1222948748=15, srv1101052629=7, srv48728489=73, srv597304922=84, srv889324366=93, srv244207362=65, srv1621081396=34, srv104223132=3, srv1111339670=8, srv1609235040=33, srv1287352322=18, srv584867229=81, srv1641110001=35, srv889833795=94, srv542462581=78, srv1080640518=5, srv129254065=19, srv1935807631=48, srv1278629659=17, srv1026943357=1, srv1133957545=10, srv475703425=72, srv1353378411=20, srv1032341052=2, srv166272086=38, srv1690732990=40, srv1490922483=26, srv1914951070=45, srv843974300=91, srv658931274=85, srv2143522717=63, srv59161497=83, srv1875292925=44, srv390184672=69, srv684699676=86, srv732229731=87, srv1647920268=36, srv568164478=80, srv2044684766=56, srv1965988692=51, srv1607856678=32, srv552239242=79, srv1172915334=14, srv540844392=77, srv52124561=75, srv999161768=99, srv1661193335=37, srv294066485=67, srv1140188385=11, srv1836687867=43, srv102049386=0, srv1150311305=13, srv2080674738=59, srv1462507729=24, srv538727160=76, srv757694362=90, srv747395394=89, srv1064663970=4, srv1452915897=23, srv2078597288=58, srv1755455220=41, srv1085790042=6, srv1395377021=22, srv232142392=64, srv893072344=95, srv1976190577=52, srv1229616715=16, srv1942024522=49, srv1141342355=12, srv1999594557=54, srv741461827=88, srv2091259417=61, srv317535327=68, srv1489980787=25, srv448910972=71, srv996344196=97, srv1594737754=30, srv1760730239=42, srv1919616772=47, srv997826431=98, srv283235678=66, srv1996083973=53, srv1554038369=28, srv1503015092=27, srv937564302=96, srv202798712=55, srv1915338067=46, srv1607537184=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 17:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 17:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 17:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 17:53:15,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 17:53:15,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 17:53:15,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 17:53:15,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 17:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 17:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 17:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 17:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 17:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 17:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 17:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 17:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 17:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 17:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 17:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 17:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 17:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-29 17:53:15,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-29 17:53:15,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-29 17:53:15,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-29 17:53:15,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-29 17:53:15,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-29 17:53:15,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-29 17:53:15,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-29 17:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-29 17:53:15,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-29 17:53:15,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-29 17:53:15,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-29 17:53:15,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-29 17:53:15,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-29 17:53:15,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-29 17:53:15,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-29 17:53:15,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-29 17:53:15,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-29 17:53:15,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-29 17:53:15,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-29 17:53:15,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-29 17:53:15,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-29 17:53:15,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-29 17:53:15,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-29 17:53:15,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-29 17:53:15,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-29 17:53:15,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-29 17:53:15,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-29 17:53:15,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-29 17:53:15,563 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:53:15,564 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-29 17:53:15,564 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.5286072396190035E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-29 17:53:15,782 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 336 ms to try 5000 different iterations. Found a solution that moves 35 regions; Going from a computed imbalance of 5.5286072396190035E-5 to a new imbalance of 7.318314534427544E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0038888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:53:15,782 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1133957545:83 , srv1150311305:83 , srv1503015092:83 , srv1915338067:83 , srv1919616772:83 , srv202798712:83 , srv102049386:84 , srv1032341052:84 , srv104223132:84 , srv1064663970:84 , srv1080640518:84 , srv1085790042:84 , srv1101052629:84 , srv1111339670:84 , srv1127831181:84 , srv1140188385:84 , srv1141342355:84 , srv1172915334:84 , srv1222948748:84 , srv1229616715:84 , srv1278629659:84 , srv1287352322:84 , srv129254065:84 , srv1378014803:84 , srv1395377021:84 , srv1452915897:84 , srv1489980787:84 , srv1490922483:84 , srv1554038369:84 , srv1593747683:84 , srv1594737754:84 , srv1607537184:84 , srv1607856678:84 , srv1609235040:84 , srv1621081396:84 , srv1641110001:84 , srv1647920268:84 , srv1661193335:84 , srv166272086:84 , srv1666211746:84 , srv1755455220:84 , srv1760730239:84 , srv1836687867:84 , srv1914951070:84 , srv1935807631:84 , srv1942024522:84 , srv1951442903:84 , srv1965988692:84 , srv1976190577:84 , srv1996083973:84 , srv2044684766:84 , srv2062328510:84 , srv2078597288:84 , srv2080674738:84 , srv2084322036:84 , srv2091259417:84 , srv2127039386:84 , srv2143522717:84 , srv232142392:84 , srv244207362:84 , srv283235678:84 , srv294066485:84 , srv317535327:84 , srv390184672:84 , srv448183756:84 , srv448910972:84 , srv475703425:84 , srv48728489:84 , srv500573581:84 , srv52124561:84 , srv538727160:84 , srv540844392:84 , srv542462581:84 , srv552239242:84 , srv568164478:84 , srv584867229:84 , srv590812418:84 , srv59161497:84 , srv597304922:84 , srv658931274:84 , srv684699676:84 , srv732229731:84 , srv741461827:84 , srv747395394:84 , srv757694362:84 , srv843974300:84 , srv878468268:84 , srv889833795:84 , srv893072344:84 , srv996344196:84 , srv997826431:84 , srv999161768:84 , srv1026943357:85 , srv1462507729:85 , srv1690732990:85 , srv1875292925:85 , srv1999594557:85 , srv889324366:85 , srv937564302:85 , srv1353378411:683 } 2024-01-29 17:53:15,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 17:53:15,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1127831181=9, srv2127039386=62, srv1378014803=21, srv1593747683=29, srv2084322036=60, srv878468268=92, srv500573581=74, srv590812418=82, srv448183756=70, srv1666211746=39, srv1951442903=50, srv2062328510=57, srv1222948748=15, srv1101052629=7, srv48728489=73, srv597304922=84, srv889324366=93, srv244207362=65, srv1621081396=34, srv104223132=3, srv1111339670=8, srv1609235040=33, srv1287352322=18, srv584867229=81, srv1641110001=35, srv889833795=94, srv542462581=78, srv1080640518=5, srv129254065=19, srv1935807631=48, srv1278629659=17, srv1026943357=1, srv1133957545=10, srv475703425=72, srv1353378411=20, srv1032341052=2, srv166272086=38, srv1690732990=40, srv1490922483=26, srv1914951070=45, srv843974300=91, srv658931274=85, srv2143522717=63, srv59161497=83, srv1875292925=44, srv390184672=69, srv684699676=86, srv732229731=87, srv1647920268=36, srv568164478=80, srv2044684766=56, srv1965988692=51, srv1607856678=32, srv552239242=79, srv1172915334=14, srv540844392=77, srv52124561=75, srv999161768=99, srv1661193335=37, srv294066485=67, srv1140188385=11, srv1836687867=43, srv102049386=0, srv1150311305=13, srv2080674738=59, srv1462507729=24, srv538727160=76, srv757694362=90, srv747395394=89, srv1064663970=4, srv1452915897=23, srv2078597288=58, srv1755455220=41, srv1085790042=6, srv1395377021=22, srv232142392=64, srv893072344=95, srv1976190577=52, srv1229616715=16, srv1942024522=49, srv1141342355=12, srv1999594557=54, srv741461827=88, srv2091259417=61, srv317535327=68, srv1489980787=25, srv448910972=71, srv996344196=97, srv1594737754=30, srv1760730239=42, srv1919616772=47, srv997826431=98, srv283235678=66, srv1996083973=53, srv1554038369=28, srv1503015092=27, srv937564302=96, srv202798712=55, srv1915338067=46, srv1607537184=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 17:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 17:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 17:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 17:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 17:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 17:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 17:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 17:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 17:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 17:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 17:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 17:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 17:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 17:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 17:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 17:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 17:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 17:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 17:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 17:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 17:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 17:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 17:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 17:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 17:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 17:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 17:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 17:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 17:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 17:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 17:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-29 17:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-29 17:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-29 17:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-29 17:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-29 17:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-29 17:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-29 17:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-29 17:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-29 17:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-29 17:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-29 17:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-29 17:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-29 17:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-29 17:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-29 17:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-29 17:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-29 17:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-29 17:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-29 17:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-29 17:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-29 17:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-29 17:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-29 17:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-29 17:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-29 17:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-29 17:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-29 17:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-29 17:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-29 17:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-29 17:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-29 17:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-29 17:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-29 17:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-29 17:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-29 17:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-29 17:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-29 17:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-29 17:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-29 17:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-29 17:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-29 17:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-29 17:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-29 17:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-29 17:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-29 17:53:15,932 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:53:15,933 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1133957545:83 , srv1150311305:83 , srv1503015092:83 , srv1915338067:83 , srv1919616772:83 , srv202798712:83 , srv102049386:84 , srv1032341052:84 , srv104223132:84 , srv1064663970:84 , srv1080640518:84 , srv1085790042:84 , srv1101052629:84 , srv1111339670:84 , srv1127831181:84 , srv1140188385:84 , srv1141342355:84 , srv1172915334:84 , srv1222948748:84 , srv1229616715:84 , srv1278629659:84 , srv1287352322:84 , srv129254065:84 , srv1378014803:84 , srv1395377021:84 , srv1452915897:84 , srv1489980787:84 , srv1490922483:84 , srv1554038369:84 , srv1593747683:84 , srv1594737754:84 , srv1607537184:84 , srv1607856678:84 , srv1609235040:84 , srv1621081396:84 , srv1641110001:84 , srv1647920268:84 , srv1661193335:84 , srv166272086:84 , srv1666211746:84 , srv1755455220:84 , srv1760730239:84 , srv1836687867:84 , srv1914951070:84 , srv1935807631:84 , srv1942024522:84 , srv1951442903:84 , srv1965988692:84 , srv1976190577:84 , srv1996083973:84 , srv2044684766:84 , srv2062328510:84 , srv2078597288:84 , srv2080674738:84 , srv2084322036:84 , srv2091259417:84 , srv2127039386:84 , srv2143522717:84 , srv232142392:84 , srv244207362:84 , srv283235678:84 , srv294066485:84 , srv317535327:84 , srv390184672:84 , srv448183756:84 , srv448910972:84 , srv475703425:84 , srv48728489:84 , srv500573581:84 , srv52124561:84 , srv538727160:84 , srv540844392:84 , srv542462581:84 , srv552239242:84 , srv568164478:84 , srv584867229:84 , srv590812418:84 , srv59161497:84 , srv597304922:84 , srv658931274:84 , srv684699676:84 , srv732229731:84 , srv741461827:84 , srv747395394:84 , srv757694362:84 , srv843974300:84 , srv878468268:84 , srv889833795:84 , srv893072344:84 , srv996344196:84 , srv997826431:84 , srv999161768:84 , srv1026943357:85 , srv1462507729:85 , srv1690732990:85 , srv1875292925:85 , srv1999594557:85 , srv889324366:85 , srv937564302:85 , srv1353378411:683 } 2024-01-29 17:53:15,956 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=325 (was 325), ProcessCount=167 (was 167), AvailableMemoryMB=2527 (was 3412) 2024-01-29 17:53:15,964 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=325, ProcessCount=167, AvailableMemoryMB=2526 2024-01-29 17:53:15,965 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-29 17:53:15,965 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-01-29 17:53:15,980 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1033670623:84 , srv1083143681:84 , srv109823758:84 , srv1156385556:84 , srv1236024109:84 , srv1253202385:84 , srv1257448639:84 , srv151913734:84 , srv157002814:84 , srv1638763041:84 , srv1731816039:84 , srv1749987760:84 , srv1759273061:84 , srv1818947400:84 , srv1953161503:84 , srv1965107485:84 , srv2105820403:84 , srv257936718:84 , srv276760079:84 , srv294543210:84 , srv334423375:84 , srv347094537:84 , srv416829263:84 , srv453329316:84 , srv506610054:84 , srv632393352:84 , srv815434172:84 , srv884677066:84 , srv978972339:84 , srv1180696875:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-01-29 17:53:15,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 17:53:15,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv815434172=27, srv151913734=8, srv1257448639=7, srv1749987760=12, srv276760079=19, srv978972339=29, srv453329316=24, srv109823758=2, srv1953161503=15, srv506610054=25, srv2105820403=17, srv257936718=18, srv1965107485=16, srv632393352=26, srv1083143681=1, srv1818947400=14, srv416829263=23, srv1759273061=13, srv1253202385=6, srv1180696875=4, srv334423375=21, srv1156385556=3, srv157002814=9, srv347094537=22, srv884677066=28, srv1731816039=11, srv294543210=20, srv1033670623=0, srv1638763041=10, srv1236024109=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 17:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 17:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 17:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 17:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 17:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 17:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 17:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 17:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 17:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 17:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 17:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 17:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 17:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 17:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 17:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 17:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 17:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 17:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-29 17:53:16,014 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1293230971706132, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:53:16,014 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912600536120676, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1293230971706132, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-01-29 17:55:16,002 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 7003237 different iterations. Found a solution that moves 1855 regions; Going from a computed imbalance of 0.9912600536120676 to a new imbalance of 4.329662539733032E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.687037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:55:16,010 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1033670623:90 , srv1083143681:90 , srv109823758:90 , srv1156385556:90 , srv1180696875:90 , srv1236024109:90 , srv1253202385:90 , srv1257448639:90 , srv151913734:90 , srv157002814:90 , srv1638763041:90 , srv1731816039:90 , srv1749987760:90 , srv1759273061:90 , srv1818947400:90 , srv1953161503:90 , srv1965107485:90 , srv2105820403:90 , srv257936718:90 , srv276760079:90 , srv294543210:90 , srv334423375:90 , srv347094537:90 , srv416829263:90 , srv453329316:90 , srv506610054:90 , srv632393352:90 , srv815434172:90 , srv884677066:90 , srv978972339:90 } 2024-01-29 17:55:16,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-29 17:55:16,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv815434172=27, srv151913734=8, srv1257448639=7, srv1749987760=12, srv276760079=19, srv978972339=29, srv453329316=24, srv109823758=2, srv1953161503=15, srv506610054=25, srv2105820403=17, srv257936718=18, srv1965107485=16, srv632393352=26, srv1083143681=1, srv1818947400=14, srv416829263=23, srv1759273061=13, srv1253202385=6, srv1180696875=4, srv334423375=21, srv1156385556=3, srv157002814=9, srv347094537=22, srv884677066=28, srv1731816039=11, srv294543210=20, srv1033670623=0, srv1638763041=10, srv1236024109=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-29 17:55:16,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-29 17:55:16,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-29 17:55:16,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-29 17:55:16,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-29 17:55:16,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-29 17:55:16,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-29 17:55:16,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-29 17:55:16,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-29 17:55:16,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-29 17:55:16,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-29 17:55:16,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-29 17:55:16,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-29 17:55:16,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-29 17:55:16,037 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-29 17:55:16,037 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1033670623:90 , srv1083143681:90 , srv109823758:90 , srv1156385556:90 , srv1180696875:90 , srv1236024109:90 , srv1253202385:90 , srv1257448639:90 , srv151913734:90 , srv157002814:90 , srv1638763041:90 , srv1731816039:90 , srv1749987760:90 , srv1759273061:90 , srv1818947400:90 , srv1953161503:90 , srv1965107485:90 , srv2105820403:90 , srv257936718:90 , srv276760079:90 , srv294543210:90 , srv334423375:90 , srv347094537:90 , srv416829263:90 , srv453329316:90 , srv506610054:90 , srv632393352:90 , srv815434172:90 , srv884677066:90 , srv978972339:90 } 2024-01-29 17:55:16,052 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=268 (was 325), ProcessCount=167 (was 167), AvailableMemoryMB=2901 (was 2526) - AvailableMemoryMB LEAK? -