2023-07-28 11:56:00,322 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-07-28 11:56:00,554 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-28 11:56:00,643 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-28 11:56:00,695 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-28 11:56:00,701 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-28 11:56:00,712 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=114, ProcessCount=171, AvailableMemoryMB=6897 2023-07-28 11:56:00,714 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-28 11:56:00,714 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-28 11:56:00,820 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000513663:84 , srv1012046826:84 , srv1017651998:84 , srv1018228293:84 , srv1022039789:84 , srv1022412183:84 , srv1024976214:84 , srv1051340439:84 , srv1052615430:84 , srv1086977482:84 , srv11043673:84 , srv1162432454:84 , srv1165756042:84 , srv1197334824:84 , srv1220779025:84 , srv122503416:84 , srv1256515755:84 , srv127186880:84 , srv1284343869:84 , srv1302852233:84 , srv1310205424:84 , srv132473638:84 , srv1345156723:84 , srv1357231298:84 , srv1359393040:84 , srv1371866735:84 , srv1393067021:84 , srv1398478441:84 , srv1466653290:84 , srv1477383880:84 , srv1495989212:84 , srv1507561957:84 , srv1528400155:84 , srv1604462533:84 , srv1644516947:84 , srv1670600551:84 , srv1706253335:84 , srv1710522895:84 , srv1714462376:84 , srv1723026188:84 , srv1735185827:84 , srv1743237086:84 , srv1764175371:84 , srv1781127744:84 , srv1784466274:84 , srv1792708442:84 , srv1873871309:84 , srv1881179198:84 , srv1935932658:84 , srv1942884268:84 , srv1977204073:84 , srv2012493676:84 , srv2014414454:84 , srv2019917451:84 , srv2045543788:84 , srv2059081964:84 , srv2065312177:84 , srv210913460:84 , srv236205238:84 , srv267752677:84 , srv295391814:84 , srv295706202:84 , srv351013249:84 , srv377068959:84 , srv3864971:84 , srv404948492:84 , srv407369142:84 , srv426311857:84 , srv450157459:84 , srv52551496:84 , srv53322746:84 , srv566011530:84 , srv570045820:84 , srv589321873:84 , srv594464769:84 , srv60505267:84 , srv618178010:84 , srv620802298:84 , srv653248283:84 , srv656694385:84 , srv673665808:84 , srv673855141:84 , srv732345683:84 , srv744777250:84 , srv74640521:84 , srv755916815:84 , srv768549044:84 , srv786227973:84 , srv788482963:84 , srv835790604:84 , srv864738635:84 , srv898328560:84 , srv913080615:84 , srv91377850:84 , srv930583069:84 , srv936252093:84 , srv939386901:84 , srv973320059:84 , srv993380440:84 , srv650583696:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-07-28 11:56:00,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 11:56:00,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv132473638=21, srv1942884268=49, srv650583696=78, srv426311857=67, srv1197334824=13, srv122503416=15, srv295706202=61, srv450157459=68, srv1743237086=41, srv2019917451=53, srv1345156723=22, srv1935932658=48, srv1706253335=36, srv1302852233=19, srv53322746=70, srv786227973=88, srv673855141=82, srv768549044=87, srv1873871309=46, srv2065312177=56, srv1977204073=50, srv653248283=79, srv570045820=72, srv404948492=65, srv1792708442=45, srv788482963=89, srv127186880=17, srv973320059=98, srv1477383880=29, srv594464769=74, srv1220779025=14, srv618178010=76, srv1357231298=23, srv1644516947=34, srv1723026188=39, srv864738635=91, srv1052615430=8, srv236205238=58, srv620802298=77, srv1714462376=38, srv1604462533=33, srv407369142=66, srv1012046826=1, srv1735185827=40, srv1781127744=43, srv1024976214=6, srv835790604=90, srv1017651998=2, srv732345683=83, srv1466653290=28, srv656694385=80, srv2059081964=55, srv2014414454=52, srv1710522895=37, srv1784466274=44, srv566011530=71, srv589321873=73, srv1881179198=47, srv898328560=92, srv377068959=63, srv1528400155=32, srv1495989212=30, srv744777250=84, srv1764175371=42, srv936252093=96, srv11043673=10, srv930583069=95, srv1256515755=16, srv1022412183=5, srv1022039789=4, srv267752677=59, srv913080615=93, srv1371866735=25, srv755916815=86, srv1000513663=0, srv60505267=75, srv673665808=81, srv1507561957=31, srv1359393040=24, srv2012493676=51, srv295391814=60, srv1284343869=18, srv993380440=99, srv1162432454=11, srv1051340439=7, srv1398478441=27, srv1670600551=35, srv210913460=57, srv351013249=62, srv1393067021=26, srv939386901=97, srv2045543788=54, srv74640521=85, srv1310205424=20, srv52551496=69, srv1086977482=9, srv91377850=94, srv1018228293=3, srv3864971=64, srv1165756042=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 11:56:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 11:56:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 11:56:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 11:56:00,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 11:56:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 11:56:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 11:56:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 11:56:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 11:56:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 11:56:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 11:56:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 11:56:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 11:56:00,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 11:56:00,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 11:56:00,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 11:56:00,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 11:56:00,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 11:56:00,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 11:56:00,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 11:56:00,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 11:56:00,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 11:56:00,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 11:56:00,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 11:56:00,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 11:56:00,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 11:56:00,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 11:56:00,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 11:56:00,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 11:56:00,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 11:56:00,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 11:56:00,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-28 11:56:00,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-28 11:56:00,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-28 11:56:00,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-28 11:56:00,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-28 11:56:00,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-28 11:56:00,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-28 11:56:00,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-28 11:56:00,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-28 11:56:00,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-28 11:56:00,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-28 11:56:00,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-28 11:56:00,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-28 11:56:00,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-28 11:56:00,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-28 11:56:00,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-28 11:56:00,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-28 11:56:00,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-28 11:56:00,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-28 11:56:00,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-28 11:56:00,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-28 11:56:00,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-28 11:56:00,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-28 11:56:00,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-28 11:56:00,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-28 11:56:00,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-28 11:56:00,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-28 11:56:00,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-28 11:56:00,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-28 11:56:00,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-28 11:56:00,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-28 11:56:00,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-28 11:56:00,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-28 11:56:00,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-28 11:56:00,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-28 11:56:00,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-28 11:56:00,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-28 11:56:00,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-28 11:56:00,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-28 11:56:00,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-28 11:56:00,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-28 11:56:00,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-28 11:56:00,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-28 11:56:00,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-28 11:56:00,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-28 11:56:00,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-28 11:56:00,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-28 11:56:00,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-28 11:56:00,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-28 11:56:00,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-28 11:56:00,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-28 11:56:00,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-28 11:56:00,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-28 11:56:00,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-28 11:56:00,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-28 11:56:00,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-28 11:56:00,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-28 11:56:00,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-28 11:56:00,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-28 11:56:00,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-28 11:56:00,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-28 11:56:00,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-28 11:56:00,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-28 11:56:00,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-28 11:56:00,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-28 11:56:00,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-28 11:56:00,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-28 11:56:00,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-28 11:56:00,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-28 11:56:00,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-28 11:56:00,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-28 11:56:01,043 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:56:01,043 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-28 11:56:01,043 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-28 11:56:01,285 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 356 ms to try 5000 different iterations. Found a solution that moves 4627 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1893238270723238. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03959956693524608, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054476387248759146, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5141111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03959956693524608, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18458333333333332, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2519166666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:56:01,297 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1000513663:83 , srv1024976214:83 , srv1165756042:83 , srv127186880:83 , srv1398478441:83 , srv1706253335:83 , srv1792708442:83 , srv1873871309:83 , srv1977204073:83 , srv732345683:83 , srv835790604:83 , srv1018228293:84 , srv1052615430:84 , srv1162432454:84 , srv1302852233:84 , srv1357231298:84 , srv1371866735:84 , srv1393067021:84 , srv1466653290:84 , srv1507561957:84 , srv1528400155:84 , srv1710522895:84 , srv1784466274:84 , srv1881179198:84 , srv2059081964:84 , srv52551496:84 , srv786227973:84 , srv1086977482:85 , srv1284343869:85 , srv132473638:85 , srv1359393040:85 , srv2012493676:85 , srv2019917451:85 , srv236205238:85 , srv426311857:85 , srv450157459:85 , srv566011530:85 , srv594464769:85 , srv744777250:85 , srv1022412183:86 , srv1256515755:86 , srv1670600551:86 , srv2065312177:86 , srv267752677:86 , srv295391814:86 , srv3864971:86 , srv53322746:86 , srv570045820:86 , srv653248283:86 , srv673665808:86 , srv768549044:86 , srv91377850:86 , srv1017651998:87 , srv11043673:87 , srv122503416:87 , srv1345156723:87 , srv1604462533:87 , srv1735185827:87 , srv295706202:87 , srv673855141:87 , srv864738635:87 , srv898328560:87 , srv939386901:87 , srv1012046826:88 , srv1022039789:88 , srv1197334824:88 , srv1220779025:88 , srv1477383880:88 , srv1714462376:88 , srv1935932658:88 , srv1942884268:88 , srv2045543788:88 , srv210913460:88 , srv351013249:88 , srv589321873:88 , srv60505267:88 , srv618178010:88 , srv620802298:88 , srv74640521:88 , srv913080615:88 , srv930583069:88 , srv936252093:88 , srv993380440:88 , srv1051340439:89 , srv1495989212:89 , srv404948492:89 , srv407369142:89 , srv656694385:89 , srv788482963:89 , srv1743237086:90 , srv1781127744:90 , srv2014414454:90 , srv377068959:90 , srv973320059:90 , srv1644516947:91 , srv1723026188:91 , srv1310205424:93 , srv755916815:93 , srv1764175371:94 , srv650583696:442 } 2023-07-28 11:56:01,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 11:56:01,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv132473638=21, srv1942884268=49, srv650583696=78, srv426311857=67, srv1197334824=13, srv122503416=15, srv295706202=61, srv450157459=68, srv1743237086=41, srv2019917451=53, srv1345156723=22, srv1935932658=48, srv1706253335=36, srv1302852233=19, srv53322746=70, srv786227973=88, srv673855141=82, srv768549044=87, srv1873871309=46, srv2065312177=56, srv1977204073=50, srv653248283=79, srv570045820=72, srv404948492=65, srv1792708442=45, srv788482963=89, srv127186880=17, srv973320059=98, srv1477383880=29, srv594464769=74, srv1220779025=14, srv618178010=76, srv1357231298=23, srv1644516947=34, srv1723026188=39, srv864738635=91, srv1052615430=8, srv236205238=58, srv620802298=77, srv1714462376=38, srv1604462533=33, srv407369142=66, srv1012046826=1, srv1735185827=40, srv1781127744=43, srv1024976214=6, srv835790604=90, srv1017651998=2, srv732345683=83, srv1466653290=28, srv656694385=80, srv2059081964=55, srv2014414454=52, srv1710522895=37, srv1784466274=44, srv566011530=71, srv589321873=73, srv1881179198=47, srv898328560=92, srv377068959=63, srv1528400155=32, srv1495989212=30, srv744777250=84, srv1764175371=42, srv936252093=96, srv11043673=10, srv930583069=95, srv1256515755=16, srv1022412183=5, srv1022039789=4, srv267752677=59, srv913080615=93, srv1371866735=25, srv755916815=86, srv1000513663=0, srv60505267=75, srv673665808=81, srv1507561957=31, srv1359393040=24, srv2012493676=51, srv295391814=60, srv1284343869=18, srv993380440=99, srv1162432454=11, srv1051340439=7, srv1398478441=27, srv1670600551=35, srv210913460=57, srv351013249=62, srv1393067021=26, srv939386901=97, srv2045543788=54, srv74640521=85, srv1310205424=20, srv52551496=69, srv1086977482=9, srv91377850=94, srv1018228293=3, srv3864971=64, srv1165756042=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 11:56:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 11:56:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 11:56:01,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 11:56:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 11:56:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 11:56:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 11:56:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 11:56:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 11:56:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 11:56:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 11:56:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 11:56:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 11:56:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 11:56:01,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 11:56:01,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-28 11:56:01,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-28 11:56:01,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-28 11:56:01,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-28 11:56:01,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-28 11:56:01,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-28 11:56:01,417 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03959956693524608, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054476387248759146, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03959956693524608, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18458333333333332, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2519166666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:56:01,417 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-28 11:56:01,417 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18929142812584726, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03959956693524608, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054476387248759146, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03959956693524608, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18458333333333332, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2519166666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-28 11:56:01,594 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 257 ms to try 5000 different iterations. Found a solution that moves 2719 regions; Going from a computed imbalance of 0.18929142812584726 to a new imbalance of 0.014871304275334837. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021048107297346203); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03196869961901936, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.3021111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021048107297346203); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:56:01,602 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1764175371:78 , srv755916815:79 , srv1310205424:80 , srv1051340439:81 , srv1644516947:81 , srv1743237086:81 , srv377068959:81 , srv407369142:81 , srv1723026188:82 , srv1781127744:82 , srv1935932658:82 , srv1942884268:82 , srv2014414454:82 , srv788482963:82 , srv993380440:82 , srv1012046826:83 , srv1495989212:83 , srv1714462376:83 , srv2045543788:83 , srv210913460:83 , srv351013249:83 , srv74640521:83 , srv913080615:83 , srv930583069:83 , srv973320059:83 , srv1022412183:84 , srv1220779025:84 , srv1477383880:84 , srv1735185827:84 , srv295391814:84 , srv404948492:84 , srv570045820:84 , srv60505267:84 , srv618178010:84 , srv620802298:84 , srv936252093:84 , srv1017651998:85 , srv1022039789:85 , srv1284343869:85 , srv1345156723:85 , srv1604462533:85 , srv295706202:85 , srv653248283:85 , srv939386901:85 , srv1018228293:86 , srv11043673:86 , srv1197334824:86 , srv122503416:86 , srv1256515755:86 , srv127186880:86 , srv132473638:86 , srv1507561957:86 , srv2065312177:86 , srv267752677:86 , srv589321873:86 , srv656694385:86 , srv768549044:86 , srv864738635:86 , srv898328560:86 , srv1162432454:87 , srv1393067021:87 , srv1670600551:87 , srv2012493676:87 , srv2019917451:87 , srv2059081964:87 , srv236205238:87 , srv3864971:87 , srv450157459:87 , srv53322746:87 , srv673665808:87 , srv732345683:87 , srv91377850:87 , srv1000513663:88 , srv1024976214:88 , srv1052615430:88 , srv1086977482:88 , srv1357231298:88 , srv1359393040:88 , srv1371866735:88 , srv1528400155:88 , srv1710522895:88 , srv1881179198:88 , srv426311857:88 , srv566011530:88 , srv1302852233:89 , srv1466653290:89 , srv1873871309:89 , srv52551496:89 , srv786227973:89 , srv1792708442:90 , srv594464769:90 , srv673855141:90 , srv744777250:90 , srv1165756042:91 , srv1706253335:91 , srv1784466274:91 , srv1977204073:91 , srv835790604:92 , srv1398478441:93 , srv650583696:519 } 2023-07-28 11:56:01,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 11:56:01,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv132473638=21, srv1942884268=49, srv650583696=78, srv426311857=67, srv1197334824=13, srv122503416=15, srv295706202=61, srv450157459=68, srv1743237086=41, srv2019917451=53, srv1345156723=22, srv1935932658=48, srv1706253335=36, srv1302852233=19, srv53322746=70, srv786227973=88, srv673855141=82, srv768549044=87, srv1873871309=46, srv2065312177=56, srv1977204073=50, srv653248283=79, srv570045820=72, srv404948492=65, srv1792708442=45, srv788482963=89, srv127186880=17, srv973320059=98, srv1477383880=29, srv594464769=74, srv1220779025=14, srv618178010=76, srv1357231298=23, srv1644516947=34, srv1723026188=39, srv864738635=91, srv1052615430=8, srv236205238=58, srv620802298=77, srv1714462376=38, srv1604462533=33, srv407369142=66, srv1012046826=1, srv1735185827=40, srv1781127744=43, srv1024976214=6, srv835790604=90, srv1017651998=2, srv732345683=83, srv1466653290=28, srv656694385=80, srv2059081964=55, srv2014414454=52, srv1710522895=37, srv1784466274=44, srv566011530=71, srv589321873=73, srv1881179198=47, srv898328560=92, srv377068959=63, srv1528400155=32, srv1495989212=30, srv744777250=84, srv1764175371=42, srv936252093=96, srv11043673=10, srv930583069=95, srv1256515755=16, srv1022412183=5, srv1022039789=4, srv267752677=59, srv913080615=93, srv1371866735=25, srv755916815=86, srv1000513663=0, srv60505267=75, srv673665808=81, srv1507561957=31, srv1359393040=24, srv2012493676=51, srv295391814=60, srv1284343869=18, srv993380440=99, srv1162432454=11, srv1051340439=7, srv1398478441=27, srv1670600551=35, srv210913460=57, srv351013249=62, srv1393067021=26, srv939386901=97, srv2045543788=54, srv74640521=85, srv1310205424=20, srv52551496=69, srv1086977482=9, srv91377850=94, srv1018228293=3, srv3864971=64, srv1165756042=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 11:56:01,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 11:56:01,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 11:56:01,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 11:56:01,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 11:56:01,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 11:56:01,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 11:56:01,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 11:56:01,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 11:56:01,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 11:56:01,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 11:56:01,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-28 11:56:01,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-28 11:56:01,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-28 11:56:01,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-28 11:56:01,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-28 11:56:01,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-28 11:56:01,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-28 11:56:01,716 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021048107297346203); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03196869961901936, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021048107297346203); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:56:01,716 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-28 11:56:01,716 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014852265430409444, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021048107297346203); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03196869961901936, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021048107297346203); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-28 11:56:01,883 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 242 ms to try 5000 different iterations. Found a solution that moves 957 regions; Going from a computed imbalance of 0.014852265430409444 to a new imbalance of 3.9937617306121814E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0072198476628214546); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007842444017689564); MoveCostFunction : (multiplier=7.0, imbalance=0.10633333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0072198476628214546); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:56:01,884 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv673855141:80 , srv1398478441:81 , srv1784466274:82 , srv656694385:82 , srv835790604:82 , srv1165756042:83 , srv1197334824:83 , srv1977204073:83 , srv589321873:83 , srv594464769:83 , srv744777250:83 , srv786227973:83 , srv1022039789:84 , srv1086977482:84 , srv1310205424:84 , srv1359393040:84 , srv1466653290:84 , srv1670600551:84 , srv1706253335:84 , srv1723026188:84 , srv1792708442:84 , srv404948492:84 , srv426311857:84 , srv53322746:84 , srv566011530:84 , srv673665808:84 , srv91377850:84 , srv973320059:84 , srv11043673:85 , srv1220779025:85 , srv122503416:85 , srv1256515755:85 , srv1302852233:85 , srv1345156723:85 , srv1477383880:85 , srv1644516947:85 , srv1743237086:85 , srv1764175371:85 , srv2012493676:85 , srv2014414454:85 , srv2019917451:85 , srv2065312177:85 , srv210913460:85 , srv267752677:85 , srv295706202:85 , srv3864971:85 , srv52551496:85 , srv618178010:85 , srv755916815:85 , srv864738635:85 , srv898328560:85 , srv936252093:85 , srv1000513663:86 , srv1012046826:86 , srv1017651998:86 , srv1022412183:86 , srv1024976214:86 , srv1052615430:86 , srv1162432454:86 , srv132473638:86 , srv1357231298:86 , srv1371866735:86 , srv1393067021:86 , srv1528400155:86 , srv1604462533:86 , srv1710522895:86 , srv1714462376:86 , srv1781127744:86 , srv1873871309:86 , srv1881179198:86 , srv2045543788:86 , srv2059081964:86 , srv377068959:86 , srv450157459:86 , srv60505267:86 , srv653248283:86 , srv768549044:86 , srv913080615:86 , srv1018228293:87 , srv1051340439:87 , srv127186880:87 , srv1284343869:87 , srv1495989212:87 , srv1507561957:87 , srv1935932658:87 , srv1942884268:87 , srv236205238:87 , srv295391814:87 , srv351013249:87 , srv620802298:87 , srv74640521:87 , srv788482963:87 , srv930583069:87 , srv939386901:87 , srv993380440:87 , srv1735185827:88 , srv407369142:88 , srv570045820:88 , srv732345683:88 , srv650583696:561 } 2023-07-28 11:56:01,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 11:56:01,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv132473638=21, srv1942884268=49, srv650583696=78, srv426311857=67, srv1197334824=13, srv122503416=15, srv295706202=61, srv450157459=68, srv1743237086=41, srv2019917451=53, srv1345156723=22, srv1935932658=48, srv1706253335=36, srv1302852233=19, srv53322746=70, srv786227973=88, srv673855141=82, srv768549044=87, srv1873871309=46, srv2065312177=56, srv1977204073=50, srv653248283=79, srv570045820=72, srv404948492=65, srv1792708442=45, srv788482963=89, srv127186880=17, srv973320059=98, srv1477383880=29, srv594464769=74, srv1220779025=14, srv618178010=76, srv1357231298=23, srv1644516947=34, srv1723026188=39, srv864738635=91, srv1052615430=8, srv236205238=58, srv620802298=77, srv1714462376=38, srv1604462533=33, srv407369142=66, srv1012046826=1, srv1735185827=40, srv1781127744=43, srv1024976214=6, srv835790604=90, srv1017651998=2, srv732345683=83, srv1466653290=28, srv656694385=80, srv2059081964=55, srv2014414454=52, srv1710522895=37, srv1784466274=44, srv566011530=71, srv589321873=73, srv1881179198=47, srv898328560=92, srv377068959=63, srv1528400155=32, srv1495989212=30, srv744777250=84, srv1764175371=42, srv936252093=96, srv11043673=10, srv930583069=95, srv1256515755=16, srv1022412183=5, srv1022039789=4, srv267752677=59, srv913080615=93, srv1371866735=25, srv755916815=86, srv1000513663=0, srv60505267=75, srv673665808=81, srv1507561957=31, srv1359393040=24, srv2012493676=51, srv295391814=60, srv1284343869=18, srv993380440=99, srv1162432454=11, srv1051340439=7, srv1398478441=27, srv1670600551=35, srv210913460=57, srv351013249=62, srv1393067021=26, srv939386901=97, srv2045543788=54, srv74640521=85, srv1310205424=20, srv52551496=69, srv1086977482=9, srv91377850=94, srv1018228293=3, srv3864971=64, srv1165756042=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 11:56:01,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 11:56:01,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 11:56:01,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 11:56:01,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 11:56:01,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 11:56:01,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 11:56:01,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 11:56:01,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 11:56:01,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 11:56:01,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-28 11:56:01,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-28 11:56:01,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-28 11:56:01,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-28 11:56:01,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-28 11:56:01,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-28 11:56:01,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-28 11:56:01,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-28 11:56:01,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-28 11:56:01,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-28 11:56:01,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-28 11:56:01,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-28 11:56:01,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-28 11:56:01,979 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0072198476628214546); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007842444017689564); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0072198476628214546); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:56:01,980 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-28 11:56:01,980 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.926751158366502E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0072198476628214546); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007842444017689564); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0072198476628214546); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-28 11:56:02,201 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 292 ms to try 5000 different iterations. Found a solution that moves 219 regions; Going from a computed imbalance of 3.926751158366502E-4 to a new imbalance of 2.6016701115226748E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.024333333333333332); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:56:02,202 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1017651998:83 , srv1495989212:83 , srv1735185827:83 , srv236205238:83 , srv594464769:83 , srv620802298:83 , srv939386901:83 , srv1018228293:84 , srv1052615430:84 , srv11043673:84 , srv1162432454:84 , srv122503416:84 , srv1284343869:84 , srv1302852233:84 , srv1345156723:84 , srv1357231298:84 , srv1371866735:84 , srv1528400155:84 , srv1706253335:84 , srv1710522895:84 , srv1714462376:84 , srv1764175371:84 , srv1781127744:84 , srv1873871309:84 , srv1881179198:84 , srv351013249:84 , srv3864971:84 , srv407369142:84 , srv450157459:84 , srv570045820:84 , srv60505267:84 , srv732345683:84 , srv744777250:84 , srv74640521:84 , srv768549044:84 , srv788482963:84 , srv864738635:84 , srv898328560:84 , srv993380440:84 , srv1000513663:85 , srv1012046826:85 , srv1022039789:85 , srv1024976214:85 , srv1051340439:85 , srv1086977482:85 , srv1165756042:85 , srv1197334824:85 , srv1220779025:85 , srv1256515755:85 , srv127186880:85 , srv1359393040:85 , srv1393067021:85 , srv1398478441:85 , srv1507561957:85 , srv1604462533:85 , srv1644516947:85 , srv1723026188:85 , srv1784466274:85 , srv1792708442:85 , srv1935932658:85 , srv1942884268:85 , srv1977204073:85 , srv2012493676:85 , srv2014414454:85 , srv2019917451:85 , srv2045543788:85 , srv2059081964:85 , srv267752677:85 , srv295391814:85 , srv295706202:85 , srv377068959:85 , srv404948492:85 , srv426311857:85 , srv52551496:85 , srv53322746:85 , srv566011530:85 , srv589321873:85 , srv618178010:85 , srv653248283:85 , srv656694385:85 , srv673665808:85 , srv673855141:85 , srv755916815:85 , srv835790604:85 , srv913080615:85 , srv91377850:85 , srv930583069:85 , srv936252093:85 , srv973320059:85 , srv1022412183:86 , srv1310205424:86 , srv132473638:86 , srv1466653290:86 , srv1477383880:86 , srv1670600551:86 , srv1743237086:86 , srv2065312177:86 , srv210913460:86 , srv786227973:86 , srv650583696:621 } 2023-07-28 11:56:02,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 11:56:02,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv132473638=21, srv1942884268=49, srv650583696=78, srv426311857=67, srv1197334824=13, srv122503416=15, srv295706202=61, srv450157459=68, srv1743237086=41, srv2019917451=53, srv1345156723=22, srv1935932658=48, srv1706253335=36, srv1302852233=19, srv53322746=70, srv786227973=88, srv673855141=82, srv768549044=87, srv1873871309=46, srv2065312177=56, srv1977204073=50, srv653248283=79, srv570045820=72, srv404948492=65, srv1792708442=45, srv788482963=89, srv127186880=17, srv973320059=98, srv1477383880=29, srv594464769=74, srv1220779025=14, srv618178010=76, srv1357231298=23, srv1644516947=34, srv1723026188=39, srv864738635=91, srv1052615430=8, srv236205238=58, srv620802298=77, srv1714462376=38, srv1604462533=33, srv407369142=66, srv1012046826=1, srv1735185827=40, srv1781127744=43, srv1024976214=6, srv835790604=90, srv1017651998=2, srv732345683=83, srv1466653290=28, srv656694385=80, srv2059081964=55, srv2014414454=52, srv1710522895=37, srv1784466274=44, srv566011530=71, srv589321873=73, srv1881179198=47, srv898328560=92, srv377068959=63, srv1528400155=32, srv1495989212=30, srv744777250=84, srv1764175371=42, srv936252093=96, srv11043673=10, srv930583069=95, srv1256515755=16, srv1022412183=5, srv1022039789=4, srv267752677=59, srv913080615=93, srv1371866735=25, srv755916815=86, srv1000513663=0, srv60505267=75, srv673665808=81, srv1507561957=31, srv1359393040=24, srv2012493676=51, srv295391814=60, srv1284343869=18, srv993380440=99, srv1162432454=11, srv1051340439=7, srv1398478441=27, srv1670600551=35, srv210913460=57, srv351013249=62, srv1393067021=26, srv939386901=97, srv2045543788=54, srv74640521=85, srv1310205424=20, srv52551496=69, srv1086977482=9, srv91377850=94, srv1018228293=3, srv3864971=64, srv1165756042=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 11:56:02,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 11:56:02,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 11:56:02,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 11:56:02,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 11:56:02,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 11:56:02,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 11:56:02,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 11:56:02,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-28 11:56:02,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-28 11:56:02,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-28 11:56:02,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-28 11:56:02,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-28 11:56:02,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-28 11:56:02,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-28 11:56:02,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-28 11:56:02,298 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:56:02,298 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-28 11:56:02,299 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.448323033969866E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-28 11:56:02,446 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 219 ms to try 5000 different iterations. Found a solution that moves 20 regions; Going from a computed imbalance of 2.448323033969866E-5 to a new imbalance of 6.88411204703086E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0022222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:56:02,447 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1310205424:83 , srv132473638:83 , srv1466653290:83 , srv1477383880:83 , srv1604462533:83 , srv1670600551:83 , srv2065312177:83 , srv52551496:83 , srv930583069:83 , srv1000513663:84 , srv1012046826:84 , srv1022039789:84 , srv1022412183:84 , srv1024976214:84 , srv1051340439:84 , srv1052615430:84 , srv1086977482:84 , srv11043673:84 , srv1165756042:84 , srv1197334824:84 , srv1220779025:84 , srv122503416:84 , srv1256515755:84 , srv1302852233:84 , srv1357231298:84 , srv1359393040:84 , srv1371866735:84 , srv1393067021:84 , srv1398478441:84 , srv1495989212:84 , srv1507561957:84 , srv1528400155:84 , srv1644516947:84 , srv1706253335:84 , srv1710522895:84 , srv1723026188:84 , srv1735185827:84 , srv1743237086:84 , srv1781127744:84 , srv1784466274:84 , srv1792708442:84 , srv1873871309:84 , srv1881179198:84 , srv1935932658:84 , srv1942884268:84 , srv1977204073:84 , srv2012493676:84 , srv2014414454:84 , srv2019917451:84 , srv2045543788:84 , srv2059081964:84 , srv210913460:84 , srv236205238:84 , srv267752677:84 , srv295391814:84 , srv295706202:84 , srv351013249:84 , srv377068959:84 , srv3864971:84 , srv404948492:84 , srv407369142:84 , srv426311857:84 , srv450157459:84 , srv53322746:84 , srv566011530:84 , srv570045820:84 , srv589321873:84 , srv60505267:84 , srv618178010:84 , srv620802298:84 , srv653248283:84 , srv656694385:84 , srv673665808:84 , srv673855141:84 , srv732345683:84 , srv744777250:84 , srv74640521:84 , srv755916815:84 , srv768549044:84 , srv786227973:84 , srv788482963:84 , srv835790604:84 , srv864738635:84 , srv898328560:84 , srv913080615:84 , srv91377850:84 , srv936252093:84 , srv939386901:84 , srv973320059:84 , srv1017651998:85 , srv1018228293:85 , srv1162432454:85 , srv127186880:85 , srv1284343869:85 , srv1345156723:85 , srv1714462376:85 , srv1764175371:85 , srv594464769:85 , srv993380440:85 , srv650583696:683 } 2023-07-28 11:56:02,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 11:56:02,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv132473638=21, srv1942884268=49, srv650583696=78, srv426311857=67, srv1197334824=13, srv122503416=15, srv295706202=61, srv450157459=68, srv1743237086=41, srv2019917451=53, srv1345156723=22, srv1935932658=48, srv1706253335=36, srv1302852233=19, srv53322746=70, srv786227973=88, srv673855141=82, srv768549044=87, srv1873871309=46, srv2065312177=56, srv1977204073=50, srv653248283=79, srv570045820=72, srv404948492=65, srv1792708442=45, srv788482963=89, srv127186880=17, srv973320059=98, srv1477383880=29, srv594464769=74, srv1220779025=14, srv618178010=76, srv1357231298=23, srv1644516947=34, srv1723026188=39, srv864738635=91, srv1052615430=8, srv236205238=58, srv620802298=77, srv1714462376=38, srv1604462533=33, srv407369142=66, srv1012046826=1, srv1735185827=40, srv1781127744=43, srv1024976214=6, srv835790604=90, srv1017651998=2, srv732345683=83, srv1466653290=28, srv656694385=80, srv2059081964=55, srv2014414454=52, srv1710522895=37, srv1784466274=44, srv566011530=71, srv589321873=73, srv1881179198=47, srv898328560=92, srv377068959=63, srv1528400155=32, srv1495989212=30, srv744777250=84, srv1764175371=42, srv936252093=96, srv11043673=10, srv930583069=95, srv1256515755=16, srv1022412183=5, srv1022039789=4, srv267752677=59, srv913080615=93, srv1371866735=25, srv755916815=86, srv1000513663=0, srv60505267=75, srv673665808=81, srv1507561957=31, srv1359393040=24, srv2012493676=51, srv295391814=60, srv1284343869=18, srv993380440=99, srv1162432454=11, srv1051340439=7, srv1398478441=27, srv1670600551=35, srv210913460=57, srv351013249=62, srv1393067021=26, srv939386901=97, srv2045543788=54, srv74640521=85, srv1310205424=20, srv52551496=69, srv1086977482=9, srv91377850=94, srv1018228293=3, srv3864971=64, srv1165756042=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 11:56:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 11:56:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 11:56:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 11:56:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 11:56:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 11:56:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 11:56:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 11:56:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 11:56:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 11:56:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 11:56:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-28 11:56:02,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-28 11:56:02,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-28 11:56:02,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-28 11:56:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-28 11:56:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-28 11:56:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-28 11:56:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-28 11:56:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-28 11:56:02,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-28 11:56:02,541 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:56:02,541 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1310205424:83 , srv132473638:83 , srv1466653290:83 , srv1477383880:83 , srv1604462533:83 , srv1670600551:83 , srv2065312177:83 , srv52551496:83 , srv930583069:83 , srv1000513663:84 , srv1012046826:84 , srv1022039789:84 , srv1022412183:84 , srv1024976214:84 , srv1051340439:84 , srv1052615430:84 , srv1086977482:84 , srv11043673:84 , srv1165756042:84 , srv1197334824:84 , srv1220779025:84 , srv122503416:84 , srv1256515755:84 , srv1302852233:84 , srv1357231298:84 , srv1359393040:84 , srv1371866735:84 , srv1393067021:84 , srv1398478441:84 , srv1495989212:84 , srv1507561957:84 , srv1528400155:84 , srv1644516947:84 , srv1706253335:84 , srv1710522895:84 , srv1723026188:84 , srv1735185827:84 , srv1743237086:84 , srv1781127744:84 , srv1784466274:84 , srv1792708442:84 , srv1873871309:84 , srv1881179198:84 , srv1935932658:84 , srv1942884268:84 , srv1977204073:84 , srv2012493676:84 , srv2014414454:84 , srv2019917451:84 , srv2045543788:84 , srv2059081964:84 , srv210913460:84 , srv236205238:84 , srv267752677:84 , srv295391814:84 , srv295706202:84 , srv351013249:84 , srv377068959:84 , srv3864971:84 , srv404948492:84 , srv407369142:84 , srv426311857:84 , srv450157459:84 , srv53322746:84 , srv566011530:84 , srv570045820:84 , srv589321873:84 , srv60505267:84 , srv618178010:84 , srv620802298:84 , srv653248283:84 , srv656694385:84 , srv673665808:84 , srv673855141:84 , srv732345683:84 , srv744777250:84 , srv74640521:84 , srv755916815:84 , srv768549044:84 , srv786227973:84 , srv788482963:84 , srv835790604:84 , srv864738635:84 , srv898328560:84 , srv913080615:84 , srv91377850:84 , srv936252093:84 , srv939386901:84 , srv973320059:84 , srv1017651998:85 , srv1018228293:85 , srv1162432454:85 , srv127186880:85 , srv1284343869:85 , srv1345156723:85 , srv1714462376:85 , srv1764175371:85 , srv594464769:85 , srv993380440:85 , srv650583696:683 } 2023-07-28 11:56:02,563 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=114 (was 114), ProcessCount=171 (was 171), AvailableMemoryMB=6408 (was 6897) 2023-07-28 11:56:02,569 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=114, ProcessCount=171, AvailableMemoryMB=6408 2023-07-28 11:56:02,570 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-28 11:56:02,570 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-07-28 11:56:02,578 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1088114988:84 , srv1105177512:84 , srv1169411975:84 , srv1253262071:84 , srv1606364339:84 , srv1608228406:84 , srv1615967440:84 , srv1740832303:84 , srv1763228359:84 , srv1768222994:84 , srv1791949879:84 , srv182991252:84 , srv1860131308:84 , srv1893900820:84 , srv2042438652:84 , srv2134391668:84 , srv236968306:84 , srv313629189:84 , srv332744816:84 , srv390506802:84 , srv415954040:84 , srv443140422:84 , srv455561897:84 , srv566025838:84 , srv664082173:84 , srv757962405:84 , srv816172766:84 , srv835352379:84 , srv892917116:84 , srv83507422:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-07-28 11:56:02,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 11:56:02,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1608228406=5, srv1791949879=10, srv332744816=18, srv664082173=24, srv313629189=17, srv1615967440=6, srv455561897=22, srv1169411975=2, srv390506802=19, srv835352379=28, srv2134391668=15, srv443140422=21, srv566025838=23, srv1763228359=8, srv1740832303=7, srv83507422=27, srv182991252=11, srv2042438652=14, srv415954040=20, srv1768222994=9, srv1105177512=1, srv1606364339=4, srv816172766=26, srv1860131308=12, srv892917116=29, srv1088114988=0, srv757962405=25, srv236968306=16, srv1893900820=13, srv1253262071=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 11:56:02,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 11:56:02,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 11:56:02,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 11:56:02,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 11:56:02,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 11:56:02,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 11:56:02,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 11:56:02,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 11:56:02,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 11:56:02,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 11:56:02,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 11:56:02,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-28 11:56:02,603 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1325797105207351, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:56:02,603 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912610797603005, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1325797105207351, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-07-28 11:58:02,592 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9461898 different iterations. Found a solution that moves 1849 regions; Going from a computed imbalance of 0.9912610797603005 to a new imbalance of 4.3156582404131396E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6848148148148148, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:58:02,598 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1088114988:90 , srv1105177512:90 , srv1169411975:90 , srv1253262071:90 , srv1606364339:90 , srv1608228406:90 , srv1615967440:90 , srv1740832303:90 , srv1763228359:90 , srv1768222994:90 , srv1791949879:90 , srv182991252:90 , srv1860131308:90 , srv1893900820:90 , srv2042438652:90 , srv2134391668:90 , srv236968306:90 , srv313629189:90 , srv332744816:90 , srv390506802:90 , srv415954040:90 , srv443140422:90 , srv455561897:90 , srv566025838:90 , srv664082173:90 , srv757962405:90 , srv816172766:90 , srv83507422:90 , srv835352379:90 , srv892917116:90 } 2023-07-28 11:58:02,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-28 11:58:02,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1608228406=5, srv1791949879=10, srv332744816=18, srv664082173=24, srv313629189=17, srv1615967440=6, srv455561897=22, srv1169411975=2, srv390506802=19, srv835352379=28, srv2134391668=15, srv443140422=21, srv566025838=23, srv1763228359=8, srv1740832303=7, srv83507422=27, srv182991252=11, srv2042438652=14, srv415954040=20, srv1768222994=9, srv1105177512=1, srv1606364339=4, srv816172766=26, srv1860131308=12, srv892917116=29, srv1088114988=0, srv757962405=25, srv236968306=16, srv1893900820=13, srv1253262071=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-28 11:58:02,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-28 11:58:02,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-28 11:58:02,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-28 11:58:02,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-28 11:58:02,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-28 11:58:02,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-28 11:58:02,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-28 11:58:02,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-28 11:58:02,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-28 11:58:02,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-28 11:58:02,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-28 11:58:02,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-28 11:58:02,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-28 11:58:02,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-28 11:58:02,617 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-28 11:58:02,617 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1088114988:90 , srv1105177512:90 , srv1169411975:90 , srv1253262071:90 , srv1606364339:90 , srv1608228406:90 , srv1615967440:90 , srv1740832303:90 , srv1763228359:90 , srv1768222994:90 , srv1791949879:90 , srv182991252:90 , srv1860131308:90 , srv1893900820:90 , srv2042438652:90 , srv2134391668:90 , srv236968306:90 , srv313629189:90 , srv332744816:90 , srv390506802:90 , srv415954040:90 , srv443140422:90 , srv455561897:90 , srv566025838:90 , srv664082173:90 , srv757962405:90 , srv816172766:90 , srv83507422:90 , srv835352379:90 , srv892917116:90 } 2023-07-28 11:58:02,628 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 114) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 171), AvailableMemoryMB=6675 (was 6408) - AvailableMemoryMB LEAK? -