2024-04-30 16:54:41,110 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-04-30 16:54:41,412 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-04-30 16:54:41,517 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-04-30 16:54:41,575 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-30 16:54:41,582 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-30 16:54:41,597 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=193, ProcessCount=169, AvailableMemoryMB=7432 2024-04-30 16:54:41,599 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-30 16:54:41,600 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-30 16:54:41,727 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1057809161:84 , srv1144814856:84 , srv1173633269:84 , srv1178767091:84 , srv1180381516:84 , srv1206567754:84 , srv120854039:84 , srv125428527:84 , srv1271560672:84 , srv1272922762:84 , srv1291140849:84 , srv1364309113:84 , srv1378070418:84 , srv1452121754:84 , srv1454215989:84 , srv1454451992:84 , srv1476949866:84 , srv1509105594:84 , srv1513057342:84 , srv1523196345:84 , srv1542856618:84 , srv1552673154:84 , srv1569687904:84 , srv15726155:84 , srv1635089249:84 , srv16709719:84 , srv1672680774:84 , srv1691622296:84 , srv1694856005:84 , srv1704167212:84 , srv1715190899:84 , srv1730830765:84 , srv1743604238:84 , srv1757876641:84 , srv1768144029:84 , srv1772383183:84 , srv1777480769:84 , srv1785271664:84 , srv1799963776:84 , srv1852633210:84 , srv1860292330:84 , srv1935937270:84 , srv1978148283:84 , srv1981085474:84 , srv1990335344:84 , srv2009841200:84 , srv2032506508:84 , srv2060256817:84 , srv2072593658:84 , srv2072887948:84 , srv2105088093:84 , srv2107780500:84 , srv215249659:84 , srv228449938:84 , srv231349914:84 , srv242171811:84 , srv249955725:84 , srv265388210:84 , srv280292035:84 , srv2832203:84 , srv29026859:84 , srv294627402:84 , srv317818055:84 , srv35354008:84 , srv353733301:84 , srv361842600:84 , srv363843687:84 , srv386728132:84 , srv427967806:84 , srv445483772:84 , srv459689710:84 , srv501941171:84 , srv502770876:84 , srv555236288:84 , srv579434923:84 , srv587653383:84 , srv604658939:84 , srv613473671:84 , srv635539464:84 , srv647048632:84 , srv674324059:84 , srv694692304:84 , srv702360496:84 , srv703071067:84 , srv765078232:84 , srv768888871:84 , srv771382587:84 , srv788351536:84 , srv789473116:84 , srv825131571:84 , srv828251532:84 , srv843686022:84 , srv851171869:84 , srv864037623:84 , srv871004332:84 , srv882243134:84 , srv909883680:84 , srv9712428:84 , srv992400168:84 , srv1646903705:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-04-30 16:54:41,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-30 16:54:41,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv228449938=54, srv1173633269=2, srv1772383183=36, srv828251532=91, srv1272922762=9, srv1144814856=1, srv2009841200=46, srv445483772=70, srv555236288=74, srv871004332=95, srv1743604238=33, srv2105088093=51, srv29026859=61, srv1860292330=41, srv265388210=58, srv2032506508=47, srv1777480769=37, srv353733301=65, srv280292035=59, srv1935937270=42, srv1291140849=10, srv35354008=64, srv2832203=60, srv1476949866=16, srv992400168=99, srv771382587=87, srv1852633210=40, srv647048632=80, srv16709719=26, srv249955725=57, srv125428527=7, srv2060256817=48, srv1981085474=44, srv1523196345=19, srv2072887948=50, srv1672680774=27, srv1180381516=4, srv1513057342=18, srv1542856618=20, srv1206567754=5, srv15726155=23, srv120854039=6, srv1978148283=43, srv2072593658=49, srv1715190899=31, srv1646903705=25, srv635539464=79, srv694692304=82, srv1452121754=13, srv1552673154=21, srv613473671=78, srv1785271664=38, srv1799963776=39, srv242171811=56, srv1454451992=15, srv1509105594=17, srv294627402=62, srv1704167212=30, srv1768144029=35, srv765078232=85, srv604658939=77, srv1691622296=28, srv386728132=68, srv1694856005=29, srv9712428=98, srv1364309113=11, srv1569687904=22, srv363843687=67, srv502770876=73, srv1178767091=3, srv703071067=84, srv427967806=69, srv674324059=81, srv864037623=94, srv1454215989=14, srv1757876641=34, srv788351536=88, srv231349914=55, srv1635089249=24, srv1730830765=32, srv825131571=90, srv459689710=71, srv1990335344=45, srv317818055=63, srv501941171=72, srv702360496=83, srv361842600=66, srv215249659=53, srv1057809161=0, srv2107780500=52, srv882243134=96, srv851171869=93, srv768888871=86, srv1271560672=8, srv579434923=75, srv789473116=89, srv909883680=97, srv1378070418=12, srv843686022=92, srv587653383=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-30 16:54:41,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-30 16:54:41,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-30 16:54:41,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-30 16:54:41,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-30 16:54:41,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-30 16:54:41,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-30 16:54:41,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-30 16:54:41,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-30 16:54:41,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-30 16:54:41,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-30 16:54:41,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-30 16:54:41,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-30 16:54:41,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-30 16:54:41,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-30 16:54:41,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-30 16:54:41,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-30 16:54:41,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-30 16:54:41,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-30 16:54:41,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-30 16:54:41,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-30 16:54:41,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-30 16:54:41,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-30 16:54:41,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-30 16:54:41,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-30 16:54:41,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-30 16:54:41,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-30 16:54:41,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-30 16:54:41,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-30 16:54:41,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-30 16:54:41,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-30 16:54:41,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-30 16:54:41,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-30 16:54:41,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-30 16:54:41,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-30 16:54:41,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-30 16:54:41,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-30 16:54:41,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-30 16:54:41,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-30 16:54:41,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-30 16:54:41,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-30 16:54:41,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-30 16:54:41,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-30 16:54:41,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-30 16:54:41,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-30 16:54:41,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-30 16:54:41,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-30 16:54:41,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-30 16:54:41,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-30 16:54:41,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-30 16:54:41,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-30 16:54:41,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-30 16:54:41,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-30 16:54:41,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-30 16:54:41,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-30 16:54:41,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-30 16:54:41,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-30 16:54:41,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-30 16:54:41,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-30 16:54:41,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-30 16:54:41,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-30 16:54:41,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-30 16:54:41,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-30 16:54:41,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-30 16:54:41,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-30 16:54:41,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-30 16:54:41,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-30 16:54:41,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-30 16:54:41,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-30 16:54:41,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-30 16:54:41,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-30 16:54:41,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-30 16:54:41,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-30 16:54:41,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-30 16:54:41,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-30 16:54:41,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-30 16:54:41,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-30 16:54:41,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-30 16:54:41,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-30 16:54:41,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-30 16:54:41,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-30 16:54:41,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-30 16:54:41,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-30 16:54:41,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-30 16:54:41,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-30 16:54:41,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-30 16:54:41,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-30 16:54:41,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-30 16:54:41,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-30 16:54:41,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-30 16:54:41,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-30 16:54:41,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-30 16:54:41,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-30 16:54:41,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-30 16:54:41,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-30 16:54:41,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-30 16:54:41,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-30 16:54:41,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-30 16:54:41,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-30 16:54:41,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-30 16:54:41,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-30 16:54:41,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-30 16:54:41,998 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:54:41,998 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-30 16:54:41,999 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-30 16:54:42,589 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 742 ms to try 5000 different iterations. Found a solution that moves 4682 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18044121693366816. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04028208305486209, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055353166702304064, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5202222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04028208305486209, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17591666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23983333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:54:42,599 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv120854039:81 , srv1743604238:81 , srv1378070418:82 , srv1691622296:82 , srv1715190899:82 , srv228449938:82 , srv459689710:82 , srv579434923:82 , srv702360496:82 , srv501941171:83 , srv703071067:83 , srv125428527:84 , srv1364309113:84 , srv1452121754:84 , srv1454451992:84 , srv1476949866:84 , srv15726155:84 , srv1635089249:84 , srv1730830765:84 , srv2060256817:84 , srv2107780500:84 , srv317818055:84 , srv35354008:84 , srv768888871:84 , srv825131571:84 , srv1144814856:85 , srv1291140849:85 , srv1542856618:85 , srv1785271664:85 , srv1852633210:85 , srv2072887948:85 , srv2105088093:85 , srv242171811:85 , srv280292035:85 , srv294627402:85 , srv353733301:85 , srv647048632:85 , srv828251532:85 , srv871004332:85 , srv1173633269:86 , srv1513057342:86 , srv1552673154:86 , srv16709719:86 , srv1672680774:86 , srv1772383183:86 , srv1935937270:86 , srv215249659:86 , srv386728132:86 , srv694692304:86 , srv882243134:86 , srv992400168:86 , srv1180381516:87 , srv1206567754:87 , srv1454215989:87 , srv1694856005:87 , srv1768144029:87 , srv1978148283:87 , srv249955725:87 , srv265388210:87 , srv29026859:87 , srv363843687:87 , srv427967806:87 , srv502770876:87 , srv635539464:87 , srv674324059:87 , srv1271560672:88 , srv1272922762:88 , srv1704167212:88 , srv1990335344:88 , srv2032506508:88 , srv231349914:88 , srv361842600:88 , srv445483772:88 , srv555236288:88 , srv587653383:88 , srv765078232:88 , srv788351536:88 , srv789473116:88 , srv1178767091:89 , srv1509105594:89 , srv1523196345:89 , srv1757876641:89 , srv1860292330:89 , srv1981085474:89 , srv2832203:89 , srv771382587:89 , srv9712428:89 , srv1569687904:90 , srv1777480769:90 , srv1799963776:90 , srv2009841200:90 , srv613473671:90 , srv851171869:90 , srv1057809161:91 , srv604658939:91 , srv843686022:91 , srv2072593658:92 , srv864037623:92 , srv909883680:92 , srv1646903705:448 } 2024-04-30 16:54:42,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-30 16:54:42,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv228449938=54, srv1173633269=2, srv1772383183=36, srv828251532=91, srv1272922762=9, srv1144814856=1, srv2009841200=46, srv445483772=70, srv555236288=74, srv871004332=95, srv1743604238=33, srv2105088093=51, srv29026859=61, srv1860292330=41, srv265388210=58, srv2032506508=47, srv1777480769=37, srv353733301=65, srv280292035=59, srv1935937270=42, srv1291140849=10, srv35354008=64, srv2832203=60, srv1476949866=16, srv992400168=99, srv771382587=87, srv1852633210=40, srv647048632=80, srv16709719=26, srv249955725=57, srv125428527=7, srv2060256817=48, srv1981085474=44, srv1523196345=19, srv2072887948=50, srv1672680774=27, srv1180381516=4, srv1513057342=18, srv1542856618=20, srv1206567754=5, srv15726155=23, srv120854039=6, srv1978148283=43, srv2072593658=49, srv1715190899=31, srv1646903705=25, srv635539464=79, srv694692304=82, srv1452121754=13, srv1552673154=21, srv613473671=78, srv1785271664=38, srv1799963776=39, srv242171811=56, srv1454451992=15, srv1509105594=17, srv294627402=62, srv1704167212=30, srv1768144029=35, srv765078232=85, srv604658939=77, srv1691622296=28, srv386728132=68, srv1694856005=29, srv9712428=98, srv1364309113=11, srv1569687904=22, srv363843687=67, srv502770876=73, srv1178767091=3, srv703071067=84, srv427967806=69, srv674324059=81, srv864037623=94, srv1454215989=14, srv1757876641=34, srv788351536=88, srv231349914=55, srv1635089249=24, srv1730830765=32, srv825131571=90, srv459689710=71, srv1990335344=45, srv317818055=63, srv501941171=72, srv702360496=83, srv361842600=66, srv215249659=53, srv1057809161=0, srv2107780500=52, srv882243134=96, srv851171869=93, srv768888871=86, srv1271560672=8, srv579434923=75, srv789473116=89, srv909883680=97, srv1378070418=12, srv843686022=92, srv587653383=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-30 16:54:42,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-30 16:54:42,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-30 16:54:42,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-30 16:54:42,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-30 16:54:42,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-30 16:54:42,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-30 16:54:42,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-30 16:54:42,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-30 16:54:42,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-30 16:54:42,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-30 16:54:42,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-30 16:54:42,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-30 16:54:42,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-30 16:54:42,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-30 16:54:42,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-30 16:54:42,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-30 16:54:42,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-30 16:54:42,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-30 16:54:42,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-30 16:54:42,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-30 16:54:42,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-30 16:54:42,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-30 16:54:42,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-30 16:54:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-30 16:54:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-30 16:54:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-30 16:54:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-30 16:54:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-30 16:54:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-30 16:54:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-30 16:54:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-30 16:54:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-30 16:54:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-30 16:54:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-30 16:54:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-30 16:54:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-30 16:54:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-30 16:54:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-30 16:54:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-30 16:54:42,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-30 16:54:42,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-30 16:54:42,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-30 16:54:42,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-30 16:54:42,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-30 16:54:42,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-30 16:54:42,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-30 16:54:42,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-30 16:54:42,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-30 16:54:42,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-30 16:54:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-30 16:54:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-30 16:54:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-30 16:54:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-30 16:54:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-30 16:54:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-30 16:54:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-30 16:54:42,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-30 16:54:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-30 16:54:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-30 16:54:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-30 16:54:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-30 16:54:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-30 16:54:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-30 16:54:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-30 16:54:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-30 16:54:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-30 16:54:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-30 16:54:42,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-30 16:54:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-30 16:54:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-30 16:54:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-30 16:54:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-30 16:54:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-30 16:54:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-30 16:54:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-30 16:54:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-30 16:54:42,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-30 16:54:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-30 16:54:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-30 16:54:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-30 16:54:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-30 16:54:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-30 16:54:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-30 16:54:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-30 16:54:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-30 16:54:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-30 16:54:42,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-30 16:54:42,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-30 16:54:42,771 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04028208305486209, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055353166702304064, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04028208305486209, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17591666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23983333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:54:42,771 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-30 16:54:42,771 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18040843286896033, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04028208305486209, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055353166702304064, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04028208305486209, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17591666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23983333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-30 16:54:43,021 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 365 ms to try 5000 different iterations. Found a solution that moves 2588 regions; Going from a computed imbalance of 0.18040843286896033 to a new imbalance of 0.016723625707852168. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02238046910101961); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03558079911965534, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28755555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02238046910101961); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0135); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04758333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:54:43,030 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2072593658:79 , srv909883680:79 , srv864037623:80 , srv1178767091:81 , srv1757876641:81 , srv1057809161:82 , srv1271560672:82 , srv1777480769:82 , srv1981085474:82 , srv2832203:82 , srv361842600:82 , srv427967806:82 , srv843686022:82 , srv9712428:82 , srv1180381516:83 , srv1272922762:83 , srv1509105594:83 , srv1569687904:83 , srv2009841200:83 , srv555236288:83 , srv604658939:83 , srv613473671:83 , srv788351536:83 , srv851171869:83 , srv1694856005:84 , srv1704167212:84 , srv1772383183:84 , srv1799963776:84 , srv1860292330:84 , srv1978148283:84 , srv1990335344:84 , srv445483772:84 , srv765078232:84 , srv789473116:84 , srv1206567754:85 , srv1454215989:85 , srv1672680774:85 , srv1935937270:85 , srv215249659:85 , srv231349914:85 , srv363843687:85 , srv587653383:85 , srv635539464:85 , srv674324059:85 , srv694692304:85 , srv771382587:85 , srv882243134:85 , srv1144814856:86 , srv1291140849:86 , srv1513057342:86 , srv1523196345:86 , srv15726155:86 , srv16709719:86 , srv1785271664:86 , srv2032506508:86 , srv265388210:86 , srv29026859:86 , srv294627402:86 , srv647048632:86 , srv768888871:86 , srv825131571:86 , srv871004332:86 , srv1452121754:87 , srv1635089249:87 , srv1768144029:87 , srv2060256817:87 , srv2105088093:87 , srv242171811:87 , srv249955725:87 , srv280292035:87 , srv35354008:87 , srv386728132:87 , srv828251532:87 , srv992400168:87 , srv1364309113:88 , srv1454451992:88 , srv1542856618:88 , srv1552673154:88 , srv2072887948:88 , srv2107780500:88 , srv353733301:88 , srv501941171:88 , srv502770876:88 , srv125428527:89 , srv1378070418:89 , srv1476949866:89 , srv1691622296:89 , srv1852633210:89 , srv228449938:89 , srv317818055:89 , srv1173633269:90 , srv1715190899:90 , srv1730830765:90 , srv459689710:90 , srv579434923:90 , srv703071067:90 , srv120854039:91 , srv1743604238:91 , srv702360496:91 , srv1646903705:525 } 2024-04-30 16:54:43,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-30 16:54:43,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv228449938=54, srv1173633269=2, srv1772383183=36, srv828251532=91, srv1272922762=9, srv1144814856=1, srv2009841200=46, srv445483772=70, srv555236288=74, srv871004332=95, srv1743604238=33, srv2105088093=51, srv29026859=61, srv1860292330=41, srv265388210=58, srv2032506508=47, srv1777480769=37, srv353733301=65, srv280292035=59, srv1935937270=42, srv1291140849=10, srv35354008=64, srv2832203=60, srv1476949866=16, srv992400168=99, srv771382587=87, srv1852633210=40, srv647048632=80, srv16709719=26, srv249955725=57, srv125428527=7, srv2060256817=48, srv1981085474=44, srv1523196345=19, srv2072887948=50, srv1672680774=27, srv1180381516=4, srv1513057342=18, srv1542856618=20, srv1206567754=5, srv15726155=23, srv120854039=6, srv1978148283=43, srv2072593658=49, srv1715190899=31, srv1646903705=25, srv635539464=79, srv694692304=82, srv1452121754=13, srv1552673154=21, srv613473671=78, srv1785271664=38, srv1799963776=39, srv242171811=56, srv1454451992=15, srv1509105594=17, srv294627402=62, srv1704167212=30, srv1768144029=35, srv765078232=85, srv604658939=77, srv1691622296=28, srv386728132=68, srv1694856005=29, srv9712428=98, srv1364309113=11, srv1569687904=22, srv363843687=67, srv502770876=73, srv1178767091=3, srv703071067=84, srv427967806=69, srv674324059=81, srv864037623=94, srv1454215989=14, srv1757876641=34, srv788351536=88, srv231349914=55, srv1635089249=24, srv1730830765=32, srv825131571=90, srv459689710=71, srv1990335344=45, srv317818055=63, srv501941171=72, srv702360496=83, srv361842600=66, srv215249659=53, srv1057809161=0, srv2107780500=52, srv882243134=96, srv851171869=93, srv768888871=86, srv1271560672=8, srv579434923=75, srv789473116=89, srv909883680=97, srv1378070418=12, srv843686022=92, srv587653383=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-30 16:54:43,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-30 16:54:43,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-30 16:54:43,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-30 16:54:43,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-30 16:54:43,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-30 16:54:43,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-30 16:54:43,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-30 16:54:43,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-30 16:54:43,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-30 16:54:43,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-30 16:54:43,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-30 16:54:43,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-30 16:54:43,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-30 16:54:43,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-30 16:54:43,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-30 16:54:43,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-30 16:54:43,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-30 16:54:43,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-30 16:54:43,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-30 16:54:43,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-30 16:54:43,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-30 16:54:43,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-30 16:54:43,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-30 16:54:43,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-30 16:54:43,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-30 16:54:43,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-30 16:54:43,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-30 16:54:43,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-30 16:54:43,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-30 16:54:43,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-30 16:54:43,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-30 16:54:43,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-30 16:54:43,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-30 16:54:43,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-30 16:54:43,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-30 16:54:43,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-30 16:54:43,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-30 16:54:43,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-30 16:54:43,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-30 16:54:43,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-30 16:54:43,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-30 16:54:43,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-30 16:54:43,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-30 16:54:43,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-30 16:54:43,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-30 16:54:43,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-30 16:54:43,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-30 16:54:43,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-30 16:54:43,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-30 16:54:43,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-30 16:54:43,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-30 16:54:43,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-30 16:54:43,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-30 16:54:43,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-30 16:54:43,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-30 16:54:43,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-30 16:54:43,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-30 16:54:43,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-30 16:54:43,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-30 16:54:43,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-30 16:54:43,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-30 16:54:43,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-30 16:54:43,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-30 16:54:43,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-30 16:54:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-30 16:54:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-30 16:54:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-30 16:54:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-30 16:54:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-30 16:54:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-30 16:54:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-30 16:54:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-30 16:54:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-30 16:54:43,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-30 16:54:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-30 16:54:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-30 16:54:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-30 16:54:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-30 16:54:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-30 16:54:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-30 16:54:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-30 16:54:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-30 16:54:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-30 16:54:43,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-30 16:54:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-30 16:54:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-30 16:54:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-30 16:54:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-30 16:54:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-30 16:54:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-30 16:54:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-30 16:54:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-30 16:54:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-30 16:54:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-30 16:54:43,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-30 16:54:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-30 16:54:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-30 16:54:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-30 16:54:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-30 16:54:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-30 16:54:43,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-30 16:54:43,191 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02238046910101961); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03558079911965534, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02238046910101961); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0135); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04758333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:54:43,191 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-30 16:54:43,191 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01670550414453223, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02238046910101961); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03558079911965534, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02238046910101961); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0135); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04758333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-30 16:54:43,419 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 334 ms to try 5000 different iterations. Found a solution that moves 980 regions; Going from a computed imbalance of 0.01670550414453223 to a new imbalance of 8.710198253248506E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006549665409178627); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008301285176284588); MoveCostFunction : (multiplier=7.0, imbalance=0.10888888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006549665409178627); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:54:43,421 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1173633269:82 , srv1523196345:82 , srv1569687904:83 , srv249955725:83 , srv502770876:83 , srv604658939:83 , srv1057809161:84 , srv125428527:84 , srv1542856618:84 , srv1552673154:84 , srv16709719:84 , srv1730830765:84 , srv1743604238:84 , srv1799963776:84 , srv1852633210:84 , srv2032506508:84 , srv2107780500:84 , srv265388210:84 , srv29026859:84 , srv317818055:84 , srv353733301:84 , srv587653383:84 , srv613473671:84 , srv771382587:84 , srv843686022:84 , srv851171869:84 , srv1206567754:85 , srv1454451992:85 , srv1476949866:85 , srv1509105594:85 , srv1513057342:85 , srv1715190899:85 , srv1768144029:85 , srv1777480769:85 , srv1860292330:85 , srv1990335344:85 , srv2009841200:85 , srv2072887948:85 , srv2105088093:85 , srv231349914:85 , srv242171811:85 , srv363843687:85 , srv386728132:85 , srv445483772:85 , srv459689710:85 , srv501941171:85 , srv579434923:85 , srv635539464:85 , srv702360496:85 , srv703071067:85 , srv828251532:85 , srv992400168:85 , srv1144814856:86 , srv120854039:86 , srv1272922762:86 , srv1291140849:86 , srv1364309113:86 , srv1452121754:86 , srv1454215989:86 , srv1635089249:86 , srv1691622296:86 , srv1704167212:86 , srv1785271664:86 , srv1935937270:86 , srv1978148283:86 , srv1981085474:86 , srv2060256817:86 , srv2072593658:86 , srv215249659:86 , srv280292035:86 , srv555236288:86 , srv674324059:86 , srv765078232:86 , srv789473116:86 , srv825131571:86 , srv1178767091:87 , srv1271560672:87 , srv1378070418:87 , srv1672680774:87 , srv1694856005:87 , srv1757876641:87 , srv228449938:87 , srv2832203:87 , srv294627402:87 , srv35354008:87 , srv361842600:87 , srv647048632:87 , srv694692304:87 , srv768888871:87 , srv788351536:87 , srv864037623:87 , srv871004332:87 , srv882243134:87 , srv909883680:87 , srv9712428:87 , srv1180381516:88 , srv1772383183:88 , srv427967806:88 , srv15726155:89 , srv1646903705:543 } 2024-04-30 16:54:43,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-30 16:54:43,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv228449938=54, srv1173633269=2, srv1772383183=36, srv828251532=91, srv1272922762=9, srv1144814856=1, srv2009841200=46, srv445483772=70, srv555236288=74, srv871004332=95, srv1743604238=33, srv2105088093=51, srv29026859=61, srv1860292330=41, srv265388210=58, srv2032506508=47, srv1777480769=37, srv353733301=65, srv280292035=59, srv1935937270=42, srv1291140849=10, srv35354008=64, srv2832203=60, srv1476949866=16, srv992400168=99, srv771382587=87, srv1852633210=40, srv647048632=80, srv16709719=26, srv249955725=57, srv125428527=7, srv2060256817=48, srv1981085474=44, srv1523196345=19, srv2072887948=50, srv1672680774=27, srv1180381516=4, srv1513057342=18, srv1542856618=20, srv1206567754=5, srv15726155=23, srv120854039=6, srv1978148283=43, srv2072593658=49, srv1715190899=31, srv1646903705=25, srv635539464=79, srv694692304=82, srv1452121754=13, srv1552673154=21, srv613473671=78, srv1785271664=38, srv1799963776=39, srv242171811=56, srv1454451992=15, srv1509105594=17, srv294627402=62, srv1704167212=30, srv1768144029=35, srv765078232=85, srv604658939=77, srv1691622296=28, srv386728132=68, srv1694856005=29, srv9712428=98, srv1364309113=11, srv1569687904=22, srv363843687=67, srv502770876=73, srv1178767091=3, srv703071067=84, srv427967806=69, srv674324059=81, srv864037623=94, srv1454215989=14, srv1757876641=34, srv788351536=88, srv231349914=55, srv1635089249=24, srv1730830765=32, srv825131571=90, srv459689710=71, srv1990335344=45, srv317818055=63, srv501941171=72, srv702360496=83, srv361842600=66, srv215249659=53, srv1057809161=0, srv2107780500=52, srv882243134=96, srv851171869=93, srv768888871=86, srv1271560672=8, srv579434923=75, srv789473116=89, srv909883680=97, srv1378070418=12, srv843686022=92, srv587653383=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-30 16:54:43,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-30 16:54:43,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-30 16:54:43,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-30 16:54:43,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-30 16:54:43,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-30 16:54:43,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-30 16:54:43,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-30 16:54:43,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-30 16:54:43,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-30 16:54:43,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-30 16:54:43,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-30 16:54:43,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-30 16:54:43,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-30 16:54:43,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-30 16:54:43,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-30 16:54:43,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-30 16:54:43,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-30 16:54:43,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-30 16:54:43,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-30 16:54:43,546 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006549665409178627); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008301285176284588); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006549665409178627); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:54:43,546 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-30 16:54:43,546 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.641577186581039E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006549665409178627); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008301285176284588); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006549665409178627); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-30 16:54:43,877 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 426 ms to try 5000 different iterations. Found a solution that moves 282 regions; Going from a computed imbalance of 8.641577186581039E-4 to a new imbalance of 3.645068149323031E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.03133333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:54:43,878 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv15726155:83 , srv1768144029:83 , srv864037623:83 , srv1144814856:84 , srv1173633269:84 , srv1180381516:84 , srv120854039:84 , srv1291140849:84 , srv1364309113:84 , srv1378070418:84 , srv1454215989:84 , srv1476949866:84 , srv1552673154:84 , srv1672680774:84 , srv1694856005:84 , srv1704167212:84 , srv1730830765:84 , srv1772383183:84 , srv1799963776:84 , srv1852633210:84 , srv1860292330:84 , srv2009841200:84 , srv2032506508:84 , srv2072593658:84 , srv2072887948:84 , srv231349914:84 , srv280292035:84 , srv2832203:84 , srv294627402:84 , srv35354008:84 , srv386728132:84 , srv502770876:84 , srv604658939:84 , srv647048632:84 , srv674324059:84 , srv694692304:84 , srv702360496:84 , srv703071067:84 , srv765078232:84 , srv771382587:84 , srv788351536:84 , srv789473116:84 , srv871004332:84 , srv882243134:84 , srv909883680:84 , srv9712428:84 , srv992400168:84 , srv1057809161:85 , srv1178767091:85 , srv1206567754:85 , srv1272922762:85 , srv1452121754:85 , srv1454451992:85 , srv1509105594:85 , srv1513057342:85 , srv1523196345:85 , srv1542856618:85 , srv1635089249:85 , srv1691622296:85 , srv1715190899:85 , srv1757876641:85 , srv1777480769:85 , srv1785271664:85 , srv1935937270:85 , srv1978148283:85 , srv1981085474:85 , srv1990335344:85 , srv2060256817:85 , srv2105088093:85 , srv215249659:85 , srv228449938:85 , srv242171811:85 , srv249955725:85 , srv265388210:85 , srv29026859:85 , srv317818055:85 , srv353733301:85 , srv361842600:85 , srv363843687:85 , srv427967806:85 , srv445483772:85 , srv459689710:85 , srv555236288:85 , srv579434923:85 , srv587653383:85 , srv613473671:85 , srv768888871:85 , srv828251532:85 , srv843686022:85 , srv851171869:85 , srv125428527:86 , srv1271560672:86 , srv1569687904:86 , srv16709719:86 , srv1743604238:86 , srv2107780500:86 , srv501941171:86 , srv635539464:86 , srv825131571:86 , srv1646903705:626 } 2024-04-30 16:54:43,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-30 16:54:43,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv228449938=54, srv1173633269=2, srv1772383183=36, srv828251532=91, srv1272922762=9, srv1144814856=1, srv2009841200=46, srv445483772=70, srv555236288=74, srv871004332=95, srv1743604238=33, srv2105088093=51, srv29026859=61, srv1860292330=41, srv265388210=58, srv2032506508=47, srv1777480769=37, srv353733301=65, srv280292035=59, srv1935937270=42, srv1291140849=10, srv35354008=64, srv2832203=60, srv1476949866=16, srv992400168=99, srv771382587=87, srv1852633210=40, srv647048632=80, srv16709719=26, srv249955725=57, srv125428527=7, srv2060256817=48, srv1981085474=44, srv1523196345=19, srv2072887948=50, srv1672680774=27, srv1180381516=4, srv1513057342=18, srv1542856618=20, srv1206567754=5, srv15726155=23, srv120854039=6, srv1978148283=43, srv2072593658=49, srv1715190899=31, srv1646903705=25, srv635539464=79, srv694692304=82, srv1452121754=13, srv1552673154=21, srv613473671=78, srv1785271664=38, srv1799963776=39, srv242171811=56, srv1454451992=15, srv1509105594=17, srv294627402=62, srv1704167212=30, srv1768144029=35, srv765078232=85, srv604658939=77, srv1691622296=28, srv386728132=68, srv1694856005=29, srv9712428=98, srv1364309113=11, srv1569687904=22, srv363843687=67, srv502770876=73, srv1178767091=3, srv703071067=84, srv427967806=69, srv674324059=81, srv864037623=94, srv1454215989=14, srv1757876641=34, srv788351536=88, srv231349914=55, srv1635089249=24, srv1730830765=32, srv825131571=90, srv459689710=71, srv1990335344=45, srv317818055=63, srv501941171=72, srv702360496=83, srv361842600=66, srv215249659=53, srv1057809161=0, srv2107780500=52, srv882243134=96, srv851171869=93, srv768888871=86, srv1271560672=8, srv579434923=75, srv789473116=89, srv909883680=97, srv1378070418=12, srv843686022=92, srv587653383=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-30 16:54:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-30 16:54:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-30 16:54:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-30 16:54:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-30 16:54:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-30 16:54:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-30 16:54:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-30 16:54:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-30 16:54:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-30 16:54:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-30 16:54:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-30 16:54:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-30 16:54:43,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-30 16:54:43,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-30 16:54:43,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-30 16:54:43,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-30 16:54:43,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-30 16:54:43,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-30 16:54:44,001 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:54:44,002 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-30 16:54:44,002 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.4476075289125655E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-30 16:54:44,211 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 305 ms to try 5000 different iterations. Found a solution that moves 30 regions; Going from a computed imbalance of 3.4476075289125655E-5 to a new imbalance of 6.9541335436303165E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0033333333333333335); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:54:44,212 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv125428527:83 , srv1271560672:83 , srv228449938:83 , srv635539464:83 , srv1057809161:84 , srv1173633269:84 , srv1178767091:84 , srv1180381516:84 , srv1206567754:84 , srv120854039:84 , srv1272922762:84 , srv1364309113:84 , srv1378070418:84 , srv1452121754:84 , srv1454215989:84 , srv1454451992:84 , srv1476949866:84 , srv1509105594:84 , srv1513057342:84 , srv1523196345:84 , srv1542856618:84 , srv1552673154:84 , srv1569687904:84 , srv15726155:84 , srv1635089249:84 , srv16709719:84 , srv1672680774:84 , srv1691622296:84 , srv1694856005:84 , srv1704167212:84 , srv1715190899:84 , srv1730830765:84 , srv1743604238:84 , srv1757876641:84 , srv1768144029:84 , srv1772383183:84 , srv1777480769:84 , srv1785271664:84 , srv1799963776:84 , srv1852633210:84 , srv1860292330:84 , srv1935937270:84 , srv1978148283:84 , srv1981085474:84 , srv1990335344:84 , srv2009841200:84 , srv2032506508:84 , srv2060256817:84 , srv2072887948:84 , srv2105088093:84 , srv2107780500:84 , srv215249659:84 , srv231349914:84 , srv242171811:84 , srv249955725:84 , srv265388210:84 , srv280292035:84 , srv2832203:84 , srv29026859:84 , srv294627402:84 , srv317818055:84 , srv35354008:84 , srv353733301:84 , srv361842600:84 , srv363843687:84 , srv386728132:84 , srv427967806:84 , srv445483772:84 , srv459689710:84 , srv501941171:84 , srv502770876:84 , srv555236288:84 , srv579434923:84 , srv587653383:84 , srv613473671:84 , srv647048632:84 , srv674324059:84 , srv694692304:84 , srv702360496:84 , srv703071067:84 , srv765078232:84 , srv768888871:84 , srv771382587:84 , srv788351536:84 , srv789473116:84 , srv825131571:84 , srv828251532:84 , srv843686022:84 , srv851171869:84 , srv864037623:84 , srv871004332:84 , srv882243134:84 , srv909883680:84 , srv9712428:84 , srv992400168:84 , srv1144814856:85 , srv1291140849:85 , srv2072593658:85 , srv604658939:85 , srv1646903705:684 } 2024-04-30 16:54:44,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-30 16:54:44,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv228449938=54, srv1173633269=2, srv1772383183=36, srv828251532=91, srv1272922762=9, srv1144814856=1, srv2009841200=46, srv445483772=70, srv555236288=74, srv871004332=95, srv1743604238=33, srv2105088093=51, srv29026859=61, srv1860292330=41, srv265388210=58, srv2032506508=47, srv1777480769=37, srv353733301=65, srv280292035=59, srv1935937270=42, srv1291140849=10, srv35354008=64, srv2832203=60, srv1476949866=16, srv992400168=99, srv771382587=87, srv1852633210=40, srv647048632=80, srv16709719=26, srv249955725=57, srv125428527=7, srv2060256817=48, srv1981085474=44, srv1523196345=19, srv2072887948=50, srv1672680774=27, srv1180381516=4, srv1513057342=18, srv1542856618=20, srv1206567754=5, srv15726155=23, srv120854039=6, srv1978148283=43, srv2072593658=49, srv1715190899=31, srv1646903705=25, srv635539464=79, srv694692304=82, srv1452121754=13, srv1552673154=21, srv613473671=78, srv1785271664=38, srv1799963776=39, srv242171811=56, srv1454451992=15, srv1509105594=17, srv294627402=62, srv1704167212=30, srv1768144029=35, srv765078232=85, srv604658939=77, srv1691622296=28, srv386728132=68, srv1694856005=29, srv9712428=98, srv1364309113=11, srv1569687904=22, srv363843687=67, srv502770876=73, srv1178767091=3, srv703071067=84, srv427967806=69, srv674324059=81, srv864037623=94, srv1454215989=14, srv1757876641=34, srv788351536=88, srv231349914=55, srv1635089249=24, srv1730830765=32, srv825131571=90, srv459689710=71, srv1990335344=45, srv317818055=63, srv501941171=72, srv702360496=83, srv361842600=66, srv215249659=53, srv1057809161=0, srv2107780500=52, srv882243134=96, srv851171869=93, srv768888871=86, srv1271560672=8, srv579434923=75, srv789473116=89, srv909883680=97, srv1378070418=12, srv843686022=92, srv587653383=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-30 16:54:44,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-30 16:54:44,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-30 16:54:44,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-30 16:54:44,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-30 16:54:44,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-30 16:54:44,346 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:54:44,346 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv125428527:83 , srv1271560672:83 , srv228449938:83 , srv635539464:83 , srv1057809161:84 , srv1173633269:84 , srv1178767091:84 , srv1180381516:84 , srv1206567754:84 , srv120854039:84 , srv1272922762:84 , srv1364309113:84 , srv1378070418:84 , srv1452121754:84 , srv1454215989:84 , srv1454451992:84 , srv1476949866:84 , srv1509105594:84 , srv1513057342:84 , srv1523196345:84 , srv1542856618:84 , srv1552673154:84 , srv1569687904:84 , srv15726155:84 , srv1635089249:84 , srv16709719:84 , srv1672680774:84 , srv1691622296:84 , srv1694856005:84 , srv1704167212:84 , srv1715190899:84 , srv1730830765:84 , srv1743604238:84 , srv1757876641:84 , srv1768144029:84 , srv1772383183:84 , srv1777480769:84 , srv1785271664:84 , srv1799963776:84 , srv1852633210:84 , srv1860292330:84 , srv1935937270:84 , srv1978148283:84 , srv1981085474:84 , srv1990335344:84 , srv2009841200:84 , srv2032506508:84 , srv2060256817:84 , srv2072887948:84 , srv2105088093:84 , srv2107780500:84 , srv215249659:84 , srv231349914:84 , srv242171811:84 , srv249955725:84 , srv265388210:84 , srv280292035:84 , srv2832203:84 , srv29026859:84 , srv294627402:84 , srv317818055:84 , srv35354008:84 , srv353733301:84 , srv361842600:84 , srv363843687:84 , srv386728132:84 , srv427967806:84 , srv445483772:84 , srv459689710:84 , srv501941171:84 , srv502770876:84 , srv555236288:84 , srv579434923:84 , srv587653383:84 , srv613473671:84 , srv647048632:84 , srv674324059:84 , srv694692304:84 , srv702360496:84 , srv703071067:84 , srv765078232:84 , srv768888871:84 , srv771382587:84 , srv788351536:84 , srv789473116:84 , srv825131571:84 , srv828251532:84 , srv843686022:84 , srv851171869:84 , srv864037623:84 , srv871004332:84 , srv882243134:84 , srv909883680:84 , srv9712428:84 , srv992400168:84 , srv1144814856:85 , srv1291140849:85 , srv2072593658:85 , srv604658939:85 , srv1646903705:684 } 2024-04-30 16:54:44,372 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=210 (was 193) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6933 (was 7432) 2024-04-30 16:54:44,381 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=210, ProcessCount=169, AvailableMemoryMB=6932 2024-04-30 16:54:44,382 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-30 16:54:44,382 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-04-30 16:54:44,391 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1040884593:84 , srv1073647094:84 , srv1169557697:84 , srv1267394830:84 , srv1314996834:84 , srv1372428893:84 , srv1389859557:84 , srv1442721550:84 , srv1469216788:84 , srv1484826226:84 , srv1571725962:84 , srv1605505593:84 , srv1608233415:84 , srv1623913243:84 , srv1689672955:84 , srv1795997782:84 , srv1892796691:84 , srv1926996107:84 , srv1964838942:84 , srv2120551888:84 , srv243902113:84 , srv517152835:84 , srv54994689:84 , srv580165707:84 , srv611708532:84 , srv741961200:84 , srv779757642:84 , srv815007251:84 , srv964413034:84 , srv1737751038:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-04-30 16:54:44,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-30 16:54:44,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1795997782=16, srv1073647094=1, srv1608233415=12, srv1605505593=11, srv611708532=25, srv1040884593=0, srv779757642=27, srv517152835=22, srv1442721550=7, srv1372428893=5, srv1314996834=4, srv741961200=26, srv1623913243=13, srv1484826226=9, srv2120551888=20, srv815007251=28, srv964413034=29, srv580165707=24, srv1469216788=8, srv243902113=21, srv1267394830=3, srv54994689=23, srv1892796691=17, srv1689672955=14, srv1737751038=15, srv1571725962=10, srv1169557697=2, srv1389859557=6, srv1926996107=18, srv1964838942=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-30 16:54:44,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-30 16:54:44,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-30 16:54:44,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-30 16:54:44,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-30 16:54:44,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-30 16:54:44,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-30 16:54:44,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-30 16:54:44,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-30 16:54:44,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-30 16:54:44,418 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1015683057696108, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:54:44,418 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991251308167925, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1015683057696108, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-04-30 16:56:44,405 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6053773 different iterations. Found a solution that moves 1856 regions; Going from a computed imbalance of 0.991251308167925 to a new imbalance of 4.3319965896196796E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6874074074074074, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:56:44,414 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1040884593:90 , srv1073647094:90 , srv1169557697:90 , srv1267394830:90 , srv1314996834:90 , srv1372428893:90 , srv1389859557:90 , srv1442721550:90 , srv1469216788:90 , srv1484826226:90 , srv1571725962:90 , srv1605505593:90 , srv1608233415:90 , srv1623913243:90 , srv1689672955:90 , srv1737751038:90 , srv1795997782:90 , srv1892796691:90 , srv1926996107:90 , srv1964838942:90 , srv2120551888:90 , srv243902113:90 , srv517152835:90 , srv54994689:90 , srv580165707:90 , srv611708532:90 , srv741961200:90 , srv779757642:90 , srv815007251:90 , srv964413034:90 } 2024-04-30 16:56:44,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-30 16:56:44,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1795997782=16, srv1073647094=1, srv1608233415=12, srv1605505593=11, srv611708532=25, srv1040884593=0, srv779757642=27, srv517152835=22, srv1442721550=7, srv1372428893=5, srv1314996834=4, srv741961200=26, srv1623913243=13, srv1484826226=9, srv2120551888=20, srv815007251=28, srv964413034=29, srv580165707=24, srv1469216788=8, srv243902113=21, srv1267394830=3, srv54994689=23, srv1892796691=17, srv1689672955=14, srv1737751038=15, srv1571725962=10, srv1169557697=2, srv1389859557=6, srv1926996107=18, srv1964838942=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-30 16:56:44,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-30 16:56:44,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-30 16:56:44,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-30 16:56:44,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-30 16:56:44,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-30 16:56:44,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-30 16:56:44,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-30 16:56:44,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-30 16:56:44,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-30 16:56:44,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-30 16:56:44,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-30 16:56:44,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-30 16:56:44,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-30 16:56:44,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-30 16:56:44,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-30 16:56:44,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-30 16:56:44,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-30 16:56:44,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-30 16:56:44,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-30 16:56:44,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-30 16:56:44,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-30 16:56:44,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-30 16:56:44,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-30 16:56:44,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-30 16:56:44,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-30 16:56:44,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-30 16:56:44,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-30 16:56:44,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-30 16:56:44,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-30 16:56:44,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-30 16:56:44,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-30 16:56:44,441 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-30 16:56:44,441 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1040884593:90 , srv1073647094:90 , srv1169557697:90 , srv1267394830:90 , srv1314996834:90 , srv1372428893:90 , srv1389859557:90 , srv1442721550:90 , srv1469216788:90 , srv1484826226:90 , srv1571725962:90 , srv1605505593:90 , srv1608233415:90 , srv1623913243:90 , srv1689672955:90 , srv1737751038:90 , srv1795997782:90 , srv1892796691:90 , srv1926996107:90 , srv1964838942:90 , srv2120551888:90 , srv243902113:90 , srv517152835:90 , srv54994689:90 , srv580165707:90 , srv611708532:90 , srv741961200:90 , srv779757642:90 , srv815007251:90 , srv964413034:90 } 2024-04-30 16:56:44,459 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=220 (was 210) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 169), AvailableMemoryMB=7897 (was 6932) - AvailableMemoryMB LEAK? -