2023-02-26 20:54:01,066 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-02-26 20:54:01,369 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-26 20:54:01,482 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-26 20:54:01,550 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-26 20:54:01,560 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-26 20:54:01,577 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=338, ProcessCount=174, AvailableMemoryMB=8144 2023-02-26 20:54:01,580 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-26 20:54:01,580 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-26 20:54:01,704 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001203397:84 , srv1027660971:84 , srv1029212040:84 , srv1035751114:84 , srv1087372367:84 , srv1093311187:84 , srv1094304988:84 , srv1146542345:84 , srv1147677959:84 , srv1160253971:84 , srv1253077957:84 , srv1279186145:84 , srv1329405812:84 , srv1339397051:84 , srv1422469360:84 , srv1433194719:84 , srv1443248431:84 , srv1449424711:84 , srv1452597058:84 , srv1467440631:84 , srv1468500866:84 , srv1469146393:84 , srv1477446788:84 , srv1544735909:84 , srv1561058116:84 , srv160201779:84 , srv1603726954:84 , srv1616686153:84 , srv1635281842:84 , srv1681132271:84 , srv1683568310:84 , srv1697320831:84 , srv1719118116:84 , srv174071487:84 , srv1764001851:84 , srv1764270689:84 , srv1771714271:84 , srv1776845063:84 , srv1786298806:84 , srv1788258358:84 , srv1837644423:84 , srv1896028002:84 , srv1909454604:84 , srv1915198601:84 , srv1920139916:84 , srv1932732620:84 , srv1967569721:84 , srv1971686216:84 , srv1979941179:84 , srv1988192134:84 , srv1994975877:84 , srv2004784589:84 , srv2019237530:84 , srv2027737125:84 , srv20498477:84 , srv2058240022:84 , srv209084823:84 , srv2101898608:84 , srv2115519491:84 , srv2129188246:84 , srv2135859955:84 , srv222402212:84 , srv245330020:84 , srv271208331:84 , srv32827314:84 , srv349764476:84 , srv350303676:84 , srv356637358:84 , srv375785103:84 , srv379575205:84 , srv418260645:84 , srv422041350:84 , srv452718341:84 , srv461304420:84 , srv467147772:84 , srv475182355:84 , srv531780200:84 , srv554908239:84 , srv589393450:84 , srv634628303:84 , srv652962771:84 , srv668321942:84 , srv691017778:84 , srv694458187:84 , srv720226190:84 , srv730949891:84 , srv783740188:84 , srv829113773:84 , srv831196213:84 , srv83986678:84 , srv855290369:84 , srv868105395:84 , srv873822948:84 , srv899942658:84 , srv915130489:84 , srv938844844:84 , srv947406245:84 , srv952653609:84 , srv982984185:84 , srv1759881505:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-02-26 20:54:01,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-26 20:54:01,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1776845063=38, srv1443248431=16, srv1683568310=30, srv271208331=64, srv982984185=99, srv1087372367=4, srv873822948=93, srv1468500866=20, srv1561058116=24, srv694458187=84, srv1433194719=15, srv1329405812=12, srv1786298806=39, srv452718341=73, srv1971686216=48, srv1764270689=36, srv634628303=80, srv32827314=65, srv831196213=89, srv2019237530=53, srv531780200=77, srv1467440631=19, srv1988192134=50, srv1909454604=43, srv160201779=25, srv245330020=63, srv375785103=69, srv668321942=82, srv1253077957=10, srv1771714271=37, srv2101898608=58, srv209084823=57, srv1915198601=44, srv1681132271=29, srv1697320831=31, srv1719118116=32, srv1896028002=42, srv1422469360=14, srv1449424711=17, srv915130489=95, srv418260645=71, srv1967569721=47, srv947406245=97, srv350303676=67, srv1093311187=5, srv2135859955=61, srv2115519491=59, srv83986678=90, srv222402212=62, srv461304420=74, srv1452597058=18, srv1635281842=28, srv20498477=55, srv2004784589=52, srv952653609=98, srv1764001851=35, srv349764476=66, srv868105395=92, srv467147772=75, srv2058240022=56, srv829113773=88, srv1837644423=41, srv1160253971=9, srv1932732620=46, srv652962771=81, srv1094304988=6, srv422041350=72, srv691017778=83, srv1339397051=13, srv2027737125=54, srv720226190=85, srv1603726954=26, srv1469146393=21, srv1544735909=23, srv730949891=86, srv938844844=96, srv1616686153=27, srv855290369=91, srv1146542345=7, srv1029212040=2, srv1001203397=0, srv1027660971=1, srv899942658=94, srv1979941179=49, srv379575205=70, srv783740188=87, srv1477446788=22, srv1920139916=45, srv1279186145=11, srv174071487=33, srv1994975877=51, srv1788258358=40, srv1147677959=8, srv356637358=68, srv1759881505=34, srv589393450=79, srv1035751114=3, srv475182355=76, srv554908239=78, srv2129188246=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-26 20:54:01,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-26 20:54:01,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-26 20:54:01,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-26 20:54:01,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-26 20:54:01,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-26 20:54:01,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-26 20:54:01,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-26 20:54:01,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-26 20:54:01,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-26 20:54:01,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-26 20:54:01,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-26 20:54:01,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-26 20:54:01,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-26 20:54:01,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-26 20:54:01,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-26 20:54:01,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-26 20:54:01,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-26 20:54:01,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-26 20:54:01,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-26 20:54:01,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-26 20:54:01,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-26 20:54:01,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-26 20:54:01,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-26 20:54:01,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-26 20:54:01,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-26 20:54:01,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-26 20:54:01,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-26 20:54:01,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-26 20:54:01,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-26 20:54:01,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-26 20:54:01,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-26 20:54:01,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-26 20:54:01,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-26 20:54:01,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-26 20:54:01,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-26 20:54:01,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-26 20:54:01,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-26 20:54:01,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-26 20:54:01,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-26 20:54:01,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-26 20:54:01,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-26 20:54:01,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-26 20:54:01,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-26 20:54:01,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-26 20:54:01,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-26 20:54:01,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-26 20:54:01,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-26 20:54:01,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-26 20:54:01,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-26 20:54:01,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-26 20:54:01,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-26 20:54:01,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-26 20:54:01,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-26 20:54:01,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-26 20:54:01,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-26 20:54:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-26 20:54:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-26 20:54:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-26 20:54:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-26 20:54:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-26 20:54:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-26 20:54:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-26 20:54:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-26 20:54:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-26 20:54:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-26 20:54:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-26 20:54:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-26 20:54:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-26 20:54:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-26 20:54:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-26 20:54:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-26 20:54:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-26 20:54:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-26 20:54:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-26 20:54:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-26 20:54:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-26 20:54:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-26 20:54:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-26 20:54:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-26 20:54:01,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-26 20:54:01,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-26 20:54:01,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-26 20:54:01,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-26 20:54:01,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-26 20:54:01,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-26 20:54:01,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-26 20:54:01,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-26 20:54:01,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-26 20:54:01,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-26 20:54:01,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-26 20:54:01,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-26 20:54:01,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-26 20:54:01,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-26 20:54:01,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-26 20:54:01,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-26 20:54:01,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-26 20:54:01,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-26 20:54:01,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-26 20:54:01,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-26 20:54:01,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-26 20:54:01,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-26 20:54:02,073 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:54:02,073 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-26 20:54:02,073 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-26 20:54:02,368 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 509 ms to try 5000 different iterations. Found a solution that moves 4655 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18459512852242618. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03614687859999475, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.052627910557963155, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5172222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03614687859999475, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17991666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24633333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:54:02,378 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1776845063:81 , srv1467440631:82 , srv1681132271:82 , srv1719118116:82 , srv174071487:82 , srv730949891:82 , srv783740188:82 , srv1449424711:83 , srv1468500866:83 , srv1603726954:83 , srv1896028002:83 , srv1932732620:83 , srv1979941179:83 , srv1160253971:84 , srv1764001851:84 , srv2101898608:84 , srv356637358:84 , srv467147772:84 , srv868105395:84 , srv1443248431:85 , srv1561058116:85 , srv1915198601:85 , srv1971686216:85 , srv1988192134:85 , srv2058240022:85 , srv209084823:85 , srv271208331:85 , srv531780200:85 , srv634628303:85 , srv691017778:85 , srv855290369:85 , srv915130489:85 , srv1027660971:86 , srv1329405812:86 , srv1469146393:86 , srv1909454604:86 , srv2004784589:86 , srv222402212:86 , srv32827314:86 , srv375785103:86 , srv418260645:86 , srv873822948:86 , srv947406245:86 , srv1001203397:87 , srv1029212040:87 , srv1035751114:87 , srv1253077957:87 , srv1339397051:87 , srv1433194719:87 , srv1477446788:87 , srv1697320831:87 , srv1764270689:87 , srv1771714271:87 , srv1994975877:87 , srv2129188246:87 , srv349764476:87 , srv461304420:87 , srv694458187:87 , srv829113773:87 , srv83986678:87 , srv1087372367:88 , srv1093311187:88 , srv1146542345:88 , srv1279186145:88 , srv1452597058:88 , srv1683568310:88 , srv1786298806:88 , srv1837644423:88 , srv1920139916:88 , srv1967569721:88 , srv475182355:88 , srv554908239:88 , srv589393450:88 , srv668321942:88 , srv952653609:88 , srv982984185:88 , srv1422469360:89 , srv1616686153:89 , srv2019237530:89 , srv20498477:89 , srv245330020:89 , srv831196213:89 , srv899942658:89 , srv1094304988:90 , srv160201779:90 , srv1635281842:90 , srv1788258358:90 , srv2027737125:90 , srv2135859955:90 , srv422041350:90 , srv452718341:90 , srv652962771:90 , srv938844844:90 , srv2115519491:91 , srv350303676:91 , srv1544735909:92 , srv720226190:92 , srv1147677959:93 , srv379575205:94 , srv1759881505:411 } 2023-02-26 20:54:02,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-26 20:54:02,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1776845063=38, srv1443248431=16, srv1683568310=30, srv271208331=64, srv982984185=99, srv1087372367=4, srv873822948=93, srv1468500866=20, srv1561058116=24, srv694458187=84, srv1433194719=15, srv1329405812=12, srv1786298806=39, srv452718341=73, srv1971686216=48, srv1764270689=36, srv634628303=80, srv32827314=65, srv831196213=89, srv2019237530=53, srv531780200=77, srv1467440631=19, srv1988192134=50, srv1909454604=43, srv160201779=25, srv245330020=63, srv375785103=69, srv668321942=82, srv1253077957=10, srv1771714271=37, srv2101898608=58, srv209084823=57, srv1915198601=44, srv1681132271=29, srv1697320831=31, srv1719118116=32, srv1896028002=42, srv1422469360=14, srv1449424711=17, srv915130489=95, srv418260645=71, srv1967569721=47, srv947406245=97, srv350303676=67, srv1093311187=5, srv2135859955=61, srv2115519491=59, srv83986678=90, srv222402212=62, srv461304420=74, srv1452597058=18, srv1635281842=28, srv20498477=55, srv2004784589=52, srv952653609=98, srv1764001851=35, srv349764476=66, srv868105395=92, srv467147772=75, srv2058240022=56, srv829113773=88, srv1837644423=41, srv1160253971=9, srv1932732620=46, srv652962771=81, srv1094304988=6, srv422041350=72, srv691017778=83, srv1339397051=13, srv2027737125=54, srv720226190=85, srv1603726954=26, srv1469146393=21, srv1544735909=23, srv730949891=86, srv938844844=96, srv1616686153=27, srv855290369=91, srv1146542345=7, srv1029212040=2, srv1001203397=0, srv1027660971=1, srv899942658=94, srv1979941179=49, srv379575205=70, srv783740188=87, srv1477446788=22, srv1920139916=45, srv1279186145=11, srv174071487=33, srv1994975877=51, srv1788258358=40, srv1147677959=8, srv356637358=68, srv1759881505=34, srv589393450=79, srv1035751114=3, srv475182355=76, srv554908239=78, srv2129188246=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-26 20:54:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-26 20:54:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-26 20:54:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-26 20:54:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-26 20:54:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-26 20:54:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-26 20:54:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-26 20:54:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-26 20:54:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-26 20:54:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-26 20:54:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-26 20:54:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-26 20:54:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-26 20:54:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-26 20:54:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-26 20:54:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-26 20:54:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-26 20:54:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-26 20:54:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-26 20:54:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-26 20:54:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-26 20:54:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-26 20:54:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-26 20:54:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-26 20:54:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-26 20:54:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-26 20:54:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-26 20:54:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-26 20:54:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-26 20:54:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-26 20:54:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-26 20:54:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-26 20:54:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-26 20:54:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-26 20:54:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-26 20:54:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-26 20:54:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-26 20:54:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-26 20:54:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-26 20:54:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-26 20:54:02,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-26 20:54:02,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-26 20:54:02,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-26 20:54:02,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-26 20:54:02,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-26 20:54:02,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-26 20:54:02,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-26 20:54:02,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-26 20:54:02,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-26 20:54:02,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-26 20:54:02,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-26 20:54:02,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-26 20:54:02,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-26 20:54:02,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-26 20:54:02,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-26 20:54:02,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-26 20:54:02,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-26 20:54:02,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-26 20:54:02,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-26 20:54:02,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-26 20:54:02,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-26 20:54:02,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-26 20:54:02,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-26 20:54:02,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-26 20:54:02,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-26 20:54:02,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-26 20:54:02,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-26 20:54:02,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-26 20:54:02,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-26 20:54:02,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-26 20:54:02,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-26 20:54:02,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-26 20:54:02,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-26 20:54:02,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-26 20:54:02,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-26 20:54:02,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-26 20:54:02,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-26 20:54:02,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-26 20:54:02,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-26 20:54:02,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-26 20:54:02,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-26 20:54:02,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-26 20:54:02,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-26 20:54:02,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-26 20:54:02,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-26 20:54:02,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-26 20:54:02,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-26 20:54:02,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-26 20:54:02,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-26 20:54:02,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-26 20:54:02,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-26 20:54:02,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-26 20:54:02,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-26 20:54:02,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-26 20:54:02,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-26 20:54:02,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-26 20:54:02,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-26 20:54:02,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-26 20:54:02,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-26 20:54:02,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-26 20:54:02,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-26 20:54:02,569 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03614687859999475, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.052627910557963155, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03614687859999475, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17991666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24633333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:54:02,570 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-26 20:54:02,570 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18456253351575916, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03614687859999475, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.052627910557963155, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03614687859999475, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17991666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24633333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-26 20:54:02,867 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 425 ms to try 5000 different iterations. Found a solution that moves 2726 regions; Going from a computed imbalance of 0.18456253351575916 to a new imbalance of 0.0165760479712577. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018797892652517807); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02808117142414448, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.3028888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018797892652517807); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04816666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:54:02,880 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1147677959:79 , srv379575205:80 , srv452718341:80 , srv720226190:80 , srv1544735909:81 , srv1635281842:81 , srv2115519491:81 , srv350303676:81 , srv831196213:81 , srv1094304988:82 , srv1616686153:82 , srv2019237530:82 , srv2027737125:82 , srv20498477:82 , srv422041350:82 , srv938844844:82 , srv1035751114:83 , srv1093311187:83 , srv1279186145:83 , srv160201779:83 , srv1788258358:83 , srv349764476:83 , srv982984185:83 , srv1029212040:84 , srv1087372367:84 , srv1146542345:84 , srv1477446788:84 , srv1771714271:84 , srv1786298806:84 , srv1967569721:84 , srv2135859955:84 , srv554908239:84 , srv589393450:84 , srv952653609:84 , srv1001203397:85 , srv1253077957:85 , srv1329405812:85 , srv1339397051:85 , srv1422469360:85 , srv1452597058:85 , srv1697320831:85 , srv1837644423:85 , srv1920139916:85 , srv2129188246:85 , srv475182355:85 , srv531780200:85 , srv652962771:85 , srv668321942:85 , srv694458187:85 , srv899942658:85 , srv1160253971:86 , srv1469146393:86 , srv1683568310:86 , srv1764270689:86 , srv1909454604:86 , srv1988192134:86 , srv245330020:86 , srv375785103:86 , srv461304420:86 , srv829113773:86 , srv1027660971:87 , srv1433194719:87 , srv1443248431:87 , srv1994975877:87 , srv2004784589:87 , srv209084823:87 , srv271208331:87 , srv356637358:87 , srv418260645:87 , srv83986678:87 , srv868105395:87 , srv1776845063:88 , srv1896028002:88 , srv1971686216:88 , srv2058240022:88 , srv2101898608:88 , srv467147772:88 , srv634628303:88 , srv855290369:88 , srv873822948:88 , srv915130489:88 , srv947406245:88 , srv1449424711:89 , srv1468500866:89 , srv1603726954:89 , srv1681132271:89 , srv1764001851:89 , srv1915198601:89 , srv1979941179:89 , srv222402212:89 , srv32827314:89 , srv691017778:89 , srv783740188:89 , srv1467440631:90 , srv1719118116:90 , srv174071487:90 , srv1932732620:90 , srv1561058116:91 , srv730949891:92 , srv1759881505:530 } 2023-02-26 20:54:02,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-26 20:54:02,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1776845063=38, srv1443248431=16, srv1683568310=30, srv271208331=64, srv982984185=99, srv1087372367=4, srv873822948=93, srv1468500866=20, srv1561058116=24, srv694458187=84, srv1433194719=15, srv1329405812=12, srv1786298806=39, srv452718341=73, srv1971686216=48, srv1764270689=36, srv634628303=80, srv32827314=65, srv831196213=89, srv2019237530=53, srv531780200=77, srv1467440631=19, srv1988192134=50, srv1909454604=43, srv160201779=25, srv245330020=63, srv375785103=69, srv668321942=82, srv1253077957=10, srv1771714271=37, srv2101898608=58, srv209084823=57, srv1915198601=44, srv1681132271=29, srv1697320831=31, srv1719118116=32, srv1896028002=42, srv1422469360=14, srv1449424711=17, srv915130489=95, srv418260645=71, srv1967569721=47, srv947406245=97, srv350303676=67, srv1093311187=5, srv2135859955=61, srv2115519491=59, srv83986678=90, srv222402212=62, srv461304420=74, srv1452597058=18, srv1635281842=28, srv20498477=55, srv2004784589=52, srv952653609=98, srv1764001851=35, srv349764476=66, srv868105395=92, srv467147772=75, srv2058240022=56, srv829113773=88, srv1837644423=41, srv1160253971=9, srv1932732620=46, srv652962771=81, srv1094304988=6, srv422041350=72, srv691017778=83, srv1339397051=13, srv2027737125=54, srv720226190=85, srv1603726954=26, srv1469146393=21, srv1544735909=23, srv730949891=86, srv938844844=96, srv1616686153=27, srv855290369=91, srv1146542345=7, srv1029212040=2, srv1001203397=0, srv1027660971=1, srv899942658=94, srv1979941179=49, srv379575205=70, srv783740188=87, srv1477446788=22, srv1920139916=45, srv1279186145=11, srv174071487=33, srv1994975877=51, srv1788258358=40, srv1147677959=8, srv356637358=68, srv1759881505=34, srv589393450=79, srv1035751114=3, srv475182355=76, srv554908239=78, srv2129188246=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-26 20:54:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-26 20:54:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-26 20:54:02,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-26 20:54:02,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-26 20:54:02,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-26 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-26 20:54:02,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-26 20:54:02,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-26 20:54:02,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-26 20:54:02,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-26 20:54:02,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-26 20:54:03,018 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018797892652517807); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02808117142414448, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018797892652517807); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04816666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:54:03,018 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-26 20:54:03,019 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01655696011128469, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018797892652517807); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02808117142414448, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018797892652517807); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04816666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-26 20:54:03,309 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 397 ms to try 5000 different iterations. Found a solution that moves 1007 regions; Going from a computed imbalance of 0.01655696011128469 to a new imbalance of 8.89166346569875E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005090526057598092); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071852198964762574); MoveCostFunction : (multiplier=7.0, imbalance=0.11188888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005090526057598092); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:54:03,311 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1561058116:81 , srv245330020:82 , srv1422469360:83 , srv1915198601:83 , srv222402212:83 , srv32827314:83 , srv379575205:83 , srv652962771:83 , srv730949891:83 , srv873822948:83 , srv947406245:83 , srv1027660971:84 , srv1433194719:84 , srv1452597058:84 , srv1683568310:84 , srv1764001851:84 , srv1764270689:84 , srv1788258358:84 , srv1920139916:84 , srv1971686216:84 , srv1994975877:84 , srv2135859955:84 , srv461304420:84 , srv475182355:84 , srv634628303:84 , srv691017778:84 , srv829113773:84 , srv83986678:84 , srv899942658:84 , srv1146542345:85 , srv1147677959:85 , srv1339397051:85 , srv1443248431:85 , srv1449424711:85 , srv1469146393:85 , srv1544735909:85 , srv160201779:85 , srv1603726954:85 , srv1786298806:85 , srv1837644423:85 , srv1932732620:85 , srv2004784589:85 , srv2027737125:85 , srv2058240022:85 , srv209084823:85 , srv2101898608:85 , srv2115519491:85 , srv350303676:85 , srv375785103:85 , srv418260645:85 , srv554908239:85 , srv668321942:85 , srv855290369:85 , srv915130489:85 , srv1001203397:86 , srv1029212040:86 , srv1087372367:86 , srv1093311187:86 , srv1253077957:86 , srv1279186145:86 , srv1468500866:86 , srv1477446788:86 , srv1635281842:86 , srv1681132271:86 , srv1697320831:86 , srv174071487:86 , srv1896028002:86 , srv1909454604:86 , srv1967569721:86 , srv1979941179:86 , srv1988192134:86 , srv2129188246:86 , srv271208331:86 , srv422041350:86 , srv467147772:86 , srv589393450:86 , srv694458187:86 , srv720226190:86 , srv938844844:86 , srv952653609:86 , srv982984185:86 , srv1035751114:87 , srv1094304988:87 , srv1329405812:87 , srv1467440631:87 , srv1719118116:87 , srv1771714271:87 , srv2019237530:87 , srv20498477:87 , srv349764476:87 , srv356637358:87 , srv452718341:87 , srv531780200:87 , srv783740188:87 , srv831196213:87 , srv868105395:87 , srv1160253971:88 , srv1616686153:88 , srv1776845063:88 , srv1759881505:562 } 2023-02-26 20:54:03,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-26 20:54:03,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1776845063=38, srv1443248431=16, srv1683568310=30, srv271208331=64, srv982984185=99, srv1087372367=4, srv873822948=93, srv1468500866=20, srv1561058116=24, srv694458187=84, srv1433194719=15, srv1329405812=12, srv1786298806=39, srv452718341=73, srv1971686216=48, srv1764270689=36, srv634628303=80, srv32827314=65, srv831196213=89, srv2019237530=53, srv531780200=77, srv1467440631=19, srv1988192134=50, srv1909454604=43, srv160201779=25, srv245330020=63, srv375785103=69, srv668321942=82, srv1253077957=10, srv1771714271=37, srv2101898608=58, srv209084823=57, srv1915198601=44, srv1681132271=29, srv1697320831=31, srv1719118116=32, srv1896028002=42, srv1422469360=14, srv1449424711=17, srv915130489=95, srv418260645=71, srv1967569721=47, srv947406245=97, srv350303676=67, srv1093311187=5, srv2135859955=61, srv2115519491=59, srv83986678=90, srv222402212=62, srv461304420=74, srv1452597058=18, srv1635281842=28, srv20498477=55, srv2004784589=52, srv952653609=98, srv1764001851=35, srv349764476=66, srv868105395=92, srv467147772=75, srv2058240022=56, srv829113773=88, srv1837644423=41, srv1160253971=9, srv1932732620=46, srv652962771=81, srv1094304988=6, srv422041350=72, srv691017778=83, srv1339397051=13, srv2027737125=54, srv720226190=85, srv1603726954=26, srv1469146393=21, srv1544735909=23, srv730949891=86, srv938844844=96, srv1616686153=27, srv855290369=91, srv1146542345=7, srv1029212040=2, srv1001203397=0, srv1027660971=1, srv899942658=94, srv1979941179=49, srv379575205=70, srv783740188=87, srv1477446788=22, srv1920139916=45, srv1279186145=11, srv174071487=33, srv1994975877=51, srv1788258358=40, srv1147677959=8, srv356637358=68, srv1759881505=34, srv589393450=79, srv1035751114=3, srv475182355=76, srv554908239=78, srv2129188246=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-26 20:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-26 20:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-26 20:54:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-26 20:54:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-26 20:54:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-26 20:54:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-26 20:54:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-26 20:54:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-26 20:54:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-26 20:54:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-26 20:54:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-26 20:54:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-26 20:54:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-26 20:54:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-26 20:54:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-26 20:54:03,450 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005090526057598092); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071852198964762574); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005090526057598092); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:54:03,450 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-26 20:54:03,451 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.821151818623097E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005090526057598092); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071852198964762574); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005090526057598092); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-26 20:54:03,719 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 366 ms to try 5000 different iterations. Found a solution that moves 237 regions; Going from a computed imbalance of 8.821151818623097E-4 to a new imbalance of 4.166090158247718E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.026333333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:54:03,720 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1094304988:83 , srv1253077957:83 , srv1467440631:83 , srv1616686153:83 , srv1719118116:83 , srv952653609:83 , srv1001203397:84 , srv1087372367:84 , srv1160253971:84 , srv1329405812:84 , srv1433194719:84 , srv1468500866:84 , srv1544735909:84 , srv160201779:84 , srv1683568310:84 , srv1697320831:84 , srv174071487:84 , srv1771714271:84 , srv1776845063:84 , srv1896028002:84 , srv1909454604:84 , srv1932732620:84 , srv1979941179:84 , srv1988192134:84 , srv1994975877:84 , srv2004784589:84 , srv2019237530:84 , srv2027737125:84 , srv20498477:84 , srv2058240022:84 , srv2129188246:84 , srv2135859955:84 , srv222402212:84 , srv271208331:84 , srv32827314:84 , srv356637358:84 , srv418260645:84 , srv422041350:84 , srv461304420:84 , srv467147772:84 , srv589393450:84 , srv652962771:84 , srv668321942:84 , srv691017778:84 , srv694458187:84 , srv720226190:84 , srv783740188:84 , srv83986678:84 , srv855290369:84 , srv868105395:84 , srv899942658:84 , srv915130489:84 , srv938844844:84 , srv1027660971:85 , srv1029212040:85 , srv1035751114:85 , srv1093311187:85 , srv1147677959:85 , srv1279186145:85 , srv1339397051:85 , srv1422469360:85 , srv1443248431:85 , srv1449424711:85 , srv1452597058:85 , srv1469146393:85 , srv1477446788:85 , srv1561058116:85 , srv1603726954:85 , srv1635281842:85 , srv1681132271:85 , srv1764001851:85 , srv1764270689:85 , srv1786298806:85 , srv1788258358:85 , srv1837644423:85 , srv1920139916:85 , srv1967569721:85 , srv1971686216:85 , srv209084823:85 , srv2101898608:85 , srv2115519491:85 , srv245330020:85 , srv349764476:85 , srv350303676:85 , srv375785103:85 , srv379575205:85 , srv452718341:85 , srv475182355:85 , srv531780200:85 , srv554908239:85 , srv730949891:85 , srv829113773:85 , srv831196213:85 , srv873822948:85 , srv1146542345:86 , srv1915198601:86 , srv634628303:86 , srv947406245:86 , srv982984185:86 , srv1759881505:639 } 2023-02-26 20:54:03,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-26 20:54:03,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1776845063=38, srv1443248431=16, srv1683568310=30, srv271208331=64, srv982984185=99, srv1087372367=4, srv873822948=93, srv1468500866=20, srv1561058116=24, srv694458187=84, srv1433194719=15, srv1329405812=12, srv1786298806=39, srv452718341=73, srv1971686216=48, srv1764270689=36, srv634628303=80, srv32827314=65, srv831196213=89, srv2019237530=53, srv531780200=77, srv1467440631=19, srv1988192134=50, srv1909454604=43, srv160201779=25, srv245330020=63, srv375785103=69, srv668321942=82, srv1253077957=10, srv1771714271=37, srv2101898608=58, srv209084823=57, srv1915198601=44, srv1681132271=29, srv1697320831=31, srv1719118116=32, srv1896028002=42, srv1422469360=14, srv1449424711=17, srv915130489=95, srv418260645=71, srv1967569721=47, srv947406245=97, srv350303676=67, srv1093311187=5, srv2135859955=61, srv2115519491=59, srv83986678=90, srv222402212=62, srv461304420=74, srv1452597058=18, srv1635281842=28, srv20498477=55, srv2004784589=52, srv952653609=98, srv1764001851=35, srv349764476=66, srv868105395=92, srv467147772=75, srv2058240022=56, srv829113773=88, srv1837644423=41, srv1160253971=9, srv1932732620=46, srv652962771=81, srv1094304988=6, srv422041350=72, srv691017778=83, srv1339397051=13, srv2027737125=54, srv720226190=85, srv1603726954=26, srv1469146393=21, srv1544735909=23, srv730949891=86, srv938844844=96, srv1616686153=27, srv855290369=91, srv1146542345=7, srv1029212040=2, srv1001203397=0, srv1027660971=1, srv899942658=94, srv1979941179=49, srv379575205=70, srv783740188=87, srv1477446788=22, srv1920139916=45, srv1279186145=11, srv174071487=33, srv1994975877=51, srv1788258358=40, srv1147677959=8, srv356637358=68, srv1759881505=34, srv589393450=79, srv1035751114=3, srv475182355=76, srv554908239=78, srv2129188246=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-26 20:54:03,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-26 20:54:03,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-26 20:54:03,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-26 20:54:03,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-26 20:54:03,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-26 20:54:03,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-26 20:54:03,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-26 20:54:03,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-26 20:54:03,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-26 20:54:03,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-26 20:54:03,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-26 20:54:03,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-26 20:54:03,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-26 20:54:03,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-26 20:54:03,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-26 20:54:03,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-26 20:54:03,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-26 20:54:03,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-26 20:54:03,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-26 20:54:03,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-26 20:54:03,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-26 20:54:03,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-26 20:54:03,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-26 20:54:03,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-26 20:54:03,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-26 20:54:03,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-26 20:54:03,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-26 20:54:03,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-26 20:54:03,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-26 20:54:03,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-26 20:54:03,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-26 20:54:03,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-26 20:54:03,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-26 20:54:03,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-26 20:54:03,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-26 20:54:03,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-26 20:54:03,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-26 20:54:03,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-26 20:54:03,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-26 20:54:03,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-26 20:54:03,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-26 20:54:03,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-26 20:54:03,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-26 20:54:03,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-26 20:54:03,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-26 20:54:03,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-26 20:54:03,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-26 20:54:03,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-26 20:54:03,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-26 20:54:03,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-26 20:54:03,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-26 20:54:03,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-26 20:54:03,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-26 20:54:03,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-26 20:54:03,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-26 20:54:03,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-26 20:54:03,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-26 20:54:03,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-26 20:54:03,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-26 20:54:03,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-26 20:54:03,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-26 20:54:03,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-26 20:54:03,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-26 20:54:03,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-26 20:54:03,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-26 20:54:03,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-26 20:54:03,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-26 20:54:03,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-26 20:54:03,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-26 20:54:03,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-26 20:54:03,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-26 20:54:03,852 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:54:03,852 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-26 20:54:03,852 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.000139211307007E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-26 20:54:04,086 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 331 ms to try 5000 different iterations. Found a solution that moves 29 regions; Going from a computed imbalance of 4.000139211307007E-5 to a new imbalance of 5.845555333665096E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0012535023884839248); MoveCostFunction : (multiplier=7.0, imbalance=0.0032222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:54:04,086 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1146542345:83 , srv1837644423:83 , srv1915198601:83 , srv1967569721:83 , srv634628303:83 , srv947406245:83 , srv982984185:83 , srv1001203397:84 , srv1027660971:84 , srv1029212040:84 , srv1035751114:84 , srv1087372367:84 , srv1093311187:84 , srv1094304988:84 , srv1147677959:84 , srv1160253971:84 , srv1279186145:84 , srv1329405812:84 , srv1339397051:84 , srv1422469360:84 , srv1433194719:84 , srv1443248431:84 , srv1449424711:84 , srv1452597058:84 , srv1467440631:84 , srv1468500866:84 , srv1469146393:84 , srv1477446788:84 , srv1544735909:84 , srv1561058116:84 , srv160201779:84 , srv1603726954:84 , srv1616686153:84 , srv1635281842:84 , srv1681132271:84 , srv1683568310:84 , srv1697320831:84 , srv1719118116:84 , srv174071487:84 , srv1764001851:84 , srv1764270689:84 , srv1771714271:84 , srv1786298806:84 , srv1788258358:84 , srv1909454604:84 , srv1920139916:84 , srv1932732620:84 , srv1971686216:84 , srv1979941179:84 , srv1994975877:84 , srv2004784589:84 , srv2019237530:84 , srv20498477:84 , srv2058240022:84 , srv209084823:84 , srv2101898608:84 , srv2115519491:84 , srv2129188246:84 , srv2135859955:84 , srv222402212:84 , srv245330020:84 , srv271208331:84 , srv32827314:84 , srv349764476:84 , srv350303676:84 , srv356637358:84 , srv375785103:84 , srv379575205:84 , srv418260645:84 , srv422041350:84 , srv452718341:84 , srv467147772:84 , srv475182355:84 , srv531780200:84 , srv554908239:84 , srv589393450:84 , srv652962771:84 , srv668321942:84 , srv691017778:84 , srv694458187:84 , srv720226190:84 , srv730949891:84 , srv783740188:84 , srv829113773:84 , srv831196213:84 , srv83986678:84 , srv855290369:84 , srv868105395:84 , srv873822948:84 , srv899942658:84 , srv915130489:84 , srv938844844:84 , srv1253077957:85 , srv1776845063:85 , srv1896028002:85 , srv1988192134:85 , srv2027737125:85 , srv461304420:85 , srv952653609:85 , srv1759881505:684 } 2023-02-26 20:54:04,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-26 20:54:04,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1776845063=38, srv1443248431=16, srv1683568310=30, srv271208331=64, srv982984185=99, srv1087372367=4, srv873822948=93, srv1468500866=20, srv1561058116=24, srv694458187=84, srv1433194719=15, srv1329405812=12, srv1786298806=39, srv452718341=73, srv1971686216=48, srv1764270689=36, srv634628303=80, srv32827314=65, srv831196213=89, srv2019237530=53, srv531780200=77, srv1467440631=19, srv1988192134=50, srv1909454604=43, srv160201779=25, srv245330020=63, srv375785103=69, srv668321942=82, srv1253077957=10, srv1771714271=37, srv2101898608=58, srv209084823=57, srv1915198601=44, srv1681132271=29, srv1697320831=31, srv1719118116=32, srv1896028002=42, srv1422469360=14, srv1449424711=17, srv915130489=95, srv418260645=71, srv1967569721=47, srv947406245=97, srv350303676=67, srv1093311187=5, srv2135859955=61, srv2115519491=59, srv83986678=90, srv222402212=62, srv461304420=74, srv1452597058=18, srv1635281842=28, srv20498477=55, srv2004784589=52, srv952653609=98, srv1764001851=35, srv349764476=66, srv868105395=92, srv467147772=75, srv2058240022=56, srv829113773=88, srv1837644423=41, srv1160253971=9, srv1932732620=46, srv652962771=81, srv1094304988=6, srv422041350=72, srv691017778=83, srv1339397051=13, srv2027737125=54, srv720226190=85, srv1603726954=26, srv1469146393=21, srv1544735909=23, srv730949891=86, srv938844844=96, srv1616686153=27, srv855290369=91, srv1146542345=7, srv1029212040=2, srv1001203397=0, srv1027660971=1, srv899942658=94, srv1979941179=49, srv379575205=70, srv783740188=87, srv1477446788=22, srv1920139916=45, srv1279186145=11, srv174071487=33, srv1994975877=51, srv1788258358=40, srv1147677959=8, srv356637358=68, srv1759881505=34, srv589393450=79, srv1035751114=3, srv475182355=76, srv554908239=78, srv2129188246=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-26 20:54:04,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-26 20:54:04,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-26 20:54:04,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-26 20:54:04,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-26 20:54:04,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-26 20:54:04,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-26 20:54:04,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-26 20:54:04,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-26 20:54:04,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-26 20:54:04,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-26 20:54:04,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-26 20:54:04,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-26 20:54:04,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-26 20:54:04,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-26 20:54:04,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-26 20:54:04,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-26 20:54:04,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-26 20:54:04,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-26 20:54:04,206 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=5.642492993526674E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0012535023884839248); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:54:04,207 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1146542345:83 , srv1837644423:83 , srv1915198601:83 , srv1967569721:83 , srv634628303:83 , srv947406245:83 , srv982984185:83 , srv1001203397:84 , srv1027660971:84 , srv1029212040:84 , srv1035751114:84 , srv1087372367:84 , srv1093311187:84 , srv1094304988:84 , srv1147677959:84 , srv1160253971:84 , srv1279186145:84 , srv1329405812:84 , srv1339397051:84 , srv1422469360:84 , srv1433194719:84 , srv1443248431:84 , srv1449424711:84 , srv1452597058:84 , srv1467440631:84 , srv1468500866:84 , srv1469146393:84 , srv1477446788:84 , srv1544735909:84 , srv1561058116:84 , srv160201779:84 , srv1603726954:84 , srv1616686153:84 , srv1635281842:84 , srv1681132271:84 , srv1683568310:84 , srv1697320831:84 , srv1719118116:84 , srv174071487:84 , srv1764001851:84 , srv1764270689:84 , srv1771714271:84 , srv1786298806:84 , srv1788258358:84 , srv1909454604:84 , srv1920139916:84 , srv1932732620:84 , srv1971686216:84 , srv1979941179:84 , srv1994975877:84 , srv2004784589:84 , srv2019237530:84 , srv20498477:84 , srv2058240022:84 , srv209084823:84 , srv2101898608:84 , srv2115519491:84 , srv2129188246:84 , srv2135859955:84 , srv222402212:84 , srv245330020:84 , srv271208331:84 , srv32827314:84 , srv349764476:84 , srv350303676:84 , srv356637358:84 , srv375785103:84 , srv379575205:84 , srv418260645:84 , srv422041350:84 , srv452718341:84 , srv467147772:84 , srv475182355:84 , srv531780200:84 , srv554908239:84 , srv589393450:84 , srv652962771:84 , srv668321942:84 , srv691017778:84 , srv694458187:84 , srv720226190:84 , srv730949891:84 , srv783740188:84 , srv829113773:84 , srv831196213:84 , srv83986678:84 , srv855290369:84 , srv868105395:84 , srv873822948:84 , srv899942658:84 , srv915130489:84 , srv938844844:84 , srv1253077957:85 , srv1776845063:85 , srv1896028002:85 , srv1988192134:85 , srv2027737125:85 , srv461304420:85 , srv952653609:85 , srv1759881505:684 } 2023-02-26 20:54:04,229 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=343 (was 338) - SystemLoadAverage LEAK? -, ProcessCount=174 (was 174), AvailableMemoryMB=7627 (was 8144) 2023-02-26 20:54:04,238 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=343, ProcessCount=174, AvailableMemoryMB=7626 2023-02-26 20:54:04,239 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-26 20:54:04,240 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-02-26 20:54:04,252 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1099851911:84 , srv1122039226:84 , srv1193518702:84 , srv1216106386:84 , srv1216261296:84 , srv1525392153:84 , srv1559149135:84 , srv1589736937:84 , srv1646887920:84 , srv179727916:84 , srv1797299197:84 , srv1868053959:84 , srv1870778418:84 , srv1884592664:84 , srv1943617536:84 , srv2068643920:84 , srv2091966175:84 , srv2124141481:84 , srv243886445:84 , srv32407120:84 , srv32582361:84 , srv341668554:84 , srv38808597:84 , srv43560944:84 , srv442957698:84 , srv510978132:84 , srv722675551:84 , srv741997460:84 , srv924572669:84 , srv1165847310:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-02-26 20:54:04,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-26 20:54:04,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1216106386=4, srv924572669=29, srv1193518702=3, srv1099851911=0, srv1870778418=13, srv1559149135=7, srv1646887920=9, srv2124141481=18, srv442957698=25, srv741997460=28, srv32582361=21, srv1884592664=14, srv2091966175=17, srv2068643920=16, srv341668554=22, srv1868053959=12, srv1943617536=15, srv1525392153=6, srv32407120=20, srv1216261296=5, srv510978132=26, srv243886445=19, srv38808597=23, srv179727916=10, srv1165847310=2, srv1797299197=11, srv43560944=24, srv722675551=27, srv1122039226=1, srv1589736937=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-26 20:54:04,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-26 20:54:04,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-26 20:54:04,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-26 20:54:04,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-26 20:54:04,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-26 20:54:04,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-26 20:54:04,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-26 20:54:04,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-26 20:54:04,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-26 20:54:04,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-26 20:54:04,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-26 20:54:04,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-26 20:54:04,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-26 20:54:04,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-26 20:54:04,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-26 20:54:04,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-26 20:54:04,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-26 20:54:04,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-26 20:54:04,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-26 20:54:04,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-26 20:54:04,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-26 20:54:04,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-26 20:54:04,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-26 20:54:04,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-26 20:54:04,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-26 20:54:04,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-26 20:54:04,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-26 20:54:04,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-26 20:54:04,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-26 20:54:04,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-26 20:54:04,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-26 20:54:04,293 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1607734485426986, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:54:04,294 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912699635150901, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1607734485426986, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-02-26 20:56:04,273 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6411598 different iterations. Found a solution that moves 1865 regions; Going from a computed imbalance of 0.9912699635150901 to a new imbalance of 4.353003038599517E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6907407407407408, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:56:04,280 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1099851911:90 , srv1122039226:90 , srv1165847310:90 , srv1193518702:90 , srv1216106386:90 , srv1216261296:90 , srv1525392153:90 , srv1559149135:90 , srv1589736937:90 , srv1646887920:90 , srv179727916:90 , srv1797299197:90 , srv1868053959:90 , srv1870778418:90 , srv1884592664:90 , srv1943617536:90 , srv2068643920:90 , srv2091966175:90 , srv2124141481:90 , srv243886445:90 , srv32407120:90 , srv32582361:90 , srv341668554:90 , srv38808597:90 , srv43560944:90 , srv442957698:90 , srv510978132:90 , srv722675551:90 , srv741997460:90 , srv924572669:90 } 2023-02-26 20:56:04,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-26 20:56:04,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1216106386=4, srv924572669=29, srv1193518702=3, srv1099851911=0, srv1870778418=13, srv1559149135=7, srv1646887920=9, srv2124141481=18, srv442957698=25, srv741997460=28, srv32582361=21, srv1884592664=14, srv2091966175=17, srv2068643920=16, srv341668554=22, srv1868053959=12, srv1943617536=15, srv1525392153=6, srv32407120=20, srv1216261296=5, srv510978132=26, srv243886445=19, srv38808597=23, srv179727916=10, srv1165847310=2, srv1797299197=11, srv43560944=24, srv722675551=27, srv1122039226=1, srv1589736937=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-26 20:56:04,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-26 20:56:04,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-26 20:56:04,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-26 20:56:04,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-26 20:56:04,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-26 20:56:04,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-26 20:56:04,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-26 20:56:04,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-26 20:56:04,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-26 20:56:04,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-26 20:56:04,298 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-26 20:56:04,298 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1099851911:90 , srv1122039226:90 , srv1165847310:90 , srv1193518702:90 , srv1216106386:90 , srv1216261296:90 , srv1525392153:90 , srv1559149135:90 , srv1589736937:90 , srv1646887920:90 , srv179727916:90 , srv1797299197:90 , srv1868053959:90 , srv1870778418:90 , srv1884592664:90 , srv1943617536:90 , srv2068643920:90 , srv2091966175:90 , srv2124141481:90 , srv243886445:90 , srv32407120:90 , srv32582361:90 , srv341668554:90 , srv38808597:90 , srv43560944:90 , srv442957698:90 , srv510978132:90 , srv722675551:90 , srv741997460:90 , srv924572669:90 } 2023-02-26 20:56:04,312 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=227 (was 343), ProcessCount=171 (was 174), AvailableMemoryMB=7454 (was 7626)