2023-07-25 06:53:12,001 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-07-25 06:53:12,325 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-25 06:53:12,457 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-25 06:53:12,527 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-25 06:53:12,534 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-25 06:53:12,548 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=311, ProcessCount=173, AvailableMemoryMB=10389 2023-07-25 06:53:12,550 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-25 06:53:12,550 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-25 06:53:12,717 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1026723145:84 , srv1037271651:84 , srv1114077111:84 , srv1121661505:84 , srv1123067877:84 , srv112544954:84 , srv1165655453:84 , srv1169040700:84 , srv1170367310:84 , srv1183965618:84 , srv1209252603:84 , srv1248211783:84 , srv12583769:84 , srv1276325417:84 , srv1293268284:84 , srv1307313360:84 , srv1308722936:84 , srv1314112844:84 , srv1323507687:84 , srv1332721801:84 , srv1372120207:84 , srv1384533330:84 , srv1428921505:84 , srv1430939042:84 , srv1452460136:84 , srv1498240996:84 , srv1508967154:84 , srv1534531889:84 , srv1551471446:84 , srv1599818161:84 , srv1601193246:84 , srv1618582735:84 , srv1630806486:84 , srv165813791:84 , srv1683727846:84 , srv1704878715:84 , srv1709191069:84 , srv1728218454:84 , srv1733283537:84 , srv1801897985:84 , srv1806473470:84 , srv1820257232:84 , srv1827861641:84 , srv1877769491:84 , srv1879025180:84 , srv1900443636:84 , srv1939724504:84 , srv1944567722:84 , srv1961671275:84 , srv2003029351:84 , srv2043205021:84 , srv2049213004:84 , srv2049924525:84 , srv2095834026:84 , srv209971219:84 , srv213171444:84 , srv256278689:84 , srv264381210:84 , srv268381164:84 , srv280202567:84 , srv285223653:84 , srv289357427:84 , srv308878040:84 , srv313494633:84 , srv316157019:84 , srv320083917:84 , srv329926373:84 , srv356968522:84 , srv362305706:84 , srv382506991:84 , srv420524652:84 , srv483409803:84 , srv49011267:84 , srv547663075:84 , srv571968796:84 , srv584403141:84 , srv586563889:84 , srv619592279:84 , srv630097385:84 , srv63959350:84 , srv655173373:84 , srv688176378:84 , srv716082739:84 , srv74843367:84 , srv755183771:84 , srv755540294:84 , srv787688397:84 , srv801749386:84 , srv8073446:84 , srv830823169:84 , srv836824847:84 , srv849916544:84 , srv863026483:84 , srv864109712:84 , srv907638820:84 , srv918188240:84 , srv938812458:84 , srv94521834:84 , srv974181636:84 , srv594011568:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-07-25 06:53:12,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-25 06:53:12,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1801897985=39, srv1123067877=4, srv571968796=74, srv630097385=79, srv1037271651=1, srv547663075=73, srv12583769=12, srv1323507687=18, srv836824847=91, srv716082739=83, srv1307313360=15, srv316157019=64, srv1169040700=7, srv1733283537=38, srv2003029351=49, srv1630806486=32, srv586563889=76, srv907638820=95, srv2095834026=53, srv1683727846=34, srv594011568=77, srv1900443636=45, srv1428921505=22, srv285223653=60, srv8073446=89, srv356968522=67, srv209971219=54, srv1209252603=10, srv1599818161=29, srv2043205021=50, srv313494633=63, srv1248211783=11, srv1618582735=31, srv1827861641=42, srv1877769491=43, srv256278689=56, srv420524652=70, srv382506991=69, srv213171444=55, srv1114077111=2, srv1165655453=6, srv918188240=96, srv1961671275=48, srv1170367310=8, srv1452460136=24, srv1939724504=46, srv329926373=66, srv1430939042=23, srv264381210=57, srv308878040=62, srv112544954=5, srv49011267=72, srv1276325417=13, srv1498240996=25, srv1508967154=26, srv362305706=68, srv974181636=99, srv1709191069=36, srv1534531889=27, srv1704878715=35, srv1944567722=47, srv165813791=33, srv74843367=84, srv1293268284=14, srv1879025180=44, srv94521834=98, srv1332721801=19, srv755540294=86, srv864109712=94, srv2049213004=51, srv320083917=65, srv483409803=71, srv787688397=87, srv1820257232=41, srv830823169=90, srv1384533330=21, srv1372120207=20, srv280202567=59, srv801749386=88, srv1183965618=9, srv268381164=58, srv289357427=61, srv863026483=93, srv655173373=81, srv1026723145=0, srv584403141=75, srv755183771=85, srv1601193246=30, srv1728218454=37, srv63959350=80, srv688176378=82, srv938812458=97, srv1806473470=40, srv1121661505=3, srv1314112844=17, srv1551471446=28, srv849916544=92, srv2049924525=52, srv619592279=78, srv1308722936=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-25 06:53:12,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-25 06:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-25 06:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-25 06:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-25 06:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-25 06:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-25 06:53:12,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-25 06:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-25 06:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-25 06:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-25 06:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-25 06:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-25 06:53:12,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-25 06:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-25 06:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-25 06:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-25 06:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-25 06:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-25 06:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-25 06:53:12,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-25 06:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-25 06:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-25 06:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-25 06:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-25 06:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-25 06:53:12,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-25 06:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-25 06:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-25 06:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-25 06:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-25 06:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-25 06:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-25 06:53:12,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-25 06:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-25 06:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-25 06:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-25 06:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-25 06:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-25 06:53:12,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-25 06:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-25 06:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-25 06:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-25 06:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-25 06:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-25 06:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-25 06:53:12,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-25 06:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-25 06:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-25 06:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-25 06:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-25 06:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-25 06:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-25 06:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-25 06:53:12,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-25 06:53:12,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-25 06:53:12,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-25 06:53:12,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-25 06:53:12,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-25 06:53:12,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-25 06:53:12,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-25 06:53:12,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-25 06:53:12,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-25 06:53:12,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-25 06:53:12,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-25 06:53:12,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-25 06:53:12,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-25 06:53:12,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-25 06:53:12,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-25 06:53:12,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-25 06:53:12,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-25 06:53:12,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-25 06:53:12,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-25 06:53:12,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-25 06:53:12,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-25 06:53:12,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-25 06:53:12,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-25 06:53:12,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-25 06:53:12,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-25 06:53:12,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-25 06:53:12,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-25 06:53:12,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-25 06:53:12,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-25 06:53:12,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-25 06:53:12,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-25 06:53:12,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-25 06:53:12,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-25 06:53:12,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-25 06:53:12,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-25 06:53:12,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-25 06:53:12,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-25 06:53:12,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-25 06:53:12,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-25 06:53:12,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-25 06:53:12,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-25 06:53:12,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-25 06:53:12,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-25 06:53:12,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-25 06:53:12,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-25 06:53:12,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-25 06:53:12,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-25 06:53:12,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-25 06:53:13,176 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:53:13,176 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-25 06:53:13,177 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-25 06:53:13,592 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 720 ms to try 5000 different iterations. Found a solution that moves 4681 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17978788026596002. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03985289072782563, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055672603969745896, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5201111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03985289072782563, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17483333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:53:13,617 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv830823169:81 , srv849916544:81 , srv1114077111:82 , srv1293268284:82 , srv1551471446:82 , srv1728218454:82 , srv1944567722:82 , srv864109712:82 , srv1121661505:83 , srv1165655453:83 , srv1630806486:83 , srv1877769491:83 , srv2049924525:83 , srv213171444:83 , srv49011267:83 , srv571968796:83 , srv94521834:83 , srv1276325417:84 , srv2043205021:84 , srv268381164:84 , srv655173373:84 , srv688176378:84 , srv974181636:84 , srv1209252603:85 , srv1323507687:85 , srv1332721801:85 , srv1428921505:85 , srv1733283537:85 , srv1939724504:85 , srv264381210:85 , srv316157019:85 , srv356968522:85 , srv619592279:85 , srv63959350:85 , srv755183771:85 , srv918188240:85 , srv1037271651:86 , srv1170367310:86 , srv1183965618:86 , srv12583769:86 , srv1308722936:86 , srv1372120207:86 , srv1498240996:86 , srv1534531889:86 , srv1683727846:86 , srv1801897985:86 , srv1961671275:86 , srv308878040:86 , srv420524652:86 , srv586563889:86 , srv787688397:86 , srv8073446:86 , srv907638820:86 , srv1026723145:87 , srv1123067877:87 , srv1169040700:87 , srv1248211783:87 , srv1307313360:87 , srv1384533330:87 , srv1430939042:87 , srv1508967154:87 , srv1806473470:87 , srv2049213004:87 , srv2095834026:87 , srv313494633:87 , srv716082739:87 , srv863026483:87 , srv938812458:87 , srv1314112844:88 , srv1599818161:88 , srv1601193246:88 , srv1618582735:88 , srv165813791:88 , srv1820257232:88 , srv209971219:88 , srv285223653:88 , srv547663075:88 , srv630097385:88 , srv1452460136:89 , srv1704878715:89 , srv1709191069:89 , srv1879025180:89 , srv256278689:89 , srv289357427:89 , srv329926373:89 , srv483409803:89 , srv584403141:89 , srv74843367:89 , srv112544954:90 , srv1827861641:90 , srv836824847:90 , srv2003029351:91 , srv280202567:91 , srv362305706:91 , srv755540294:92 , srv801749386:92 , srv382506991:93 , srv320083917:94 , srv1900443636:95 , srv594011568:444 } 2023-07-25 06:53:13,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-25 06:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1801897985=39, srv1123067877=4, srv571968796=74, srv630097385=79, srv1037271651=1, srv547663075=73, srv12583769=12, srv1323507687=18, srv836824847=91, srv716082739=83, srv1307313360=15, srv316157019=64, srv1169040700=7, srv1733283537=38, srv2003029351=49, srv1630806486=32, srv586563889=76, srv907638820=95, srv2095834026=53, srv1683727846=34, srv594011568=77, srv1900443636=45, srv1428921505=22, srv285223653=60, srv8073446=89, srv356968522=67, srv209971219=54, srv1209252603=10, srv1599818161=29, srv2043205021=50, srv313494633=63, srv1248211783=11, srv1618582735=31, srv1827861641=42, srv1877769491=43, srv256278689=56, srv420524652=70, srv382506991=69, srv213171444=55, srv1114077111=2, srv1165655453=6, srv918188240=96, srv1961671275=48, srv1170367310=8, srv1452460136=24, srv1939724504=46, srv329926373=66, srv1430939042=23, srv264381210=57, srv308878040=62, srv112544954=5, srv49011267=72, srv1276325417=13, srv1498240996=25, srv1508967154=26, srv362305706=68, srv974181636=99, srv1709191069=36, srv1534531889=27, srv1704878715=35, srv1944567722=47, srv165813791=33, srv74843367=84, srv1293268284=14, srv1879025180=44, srv94521834=98, srv1332721801=19, srv755540294=86, srv864109712=94, srv2049213004=51, srv320083917=65, srv483409803=71, srv787688397=87, srv1820257232=41, srv830823169=90, srv1384533330=21, srv1372120207=20, srv280202567=59, srv801749386=88, srv1183965618=9, srv268381164=58, srv289357427=61, srv863026483=93, srv655173373=81, srv1026723145=0, srv584403141=75, srv755183771=85, srv1601193246=30, srv1728218454=37, srv63959350=80, srv688176378=82, srv938812458=97, srv1806473470=40, srv1121661505=3, srv1314112844=17, srv1551471446=28, srv849916544=92, srv2049924525=52, srv619592279=78, srv1308722936=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-25 06:53:13,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-25 06:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-25 06:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-25 06:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-25 06:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-25 06:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-25 06:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-25 06:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-25 06:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-25 06:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-25 06:53:13,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-25 06:53:13,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-25 06:53:13,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-25 06:53:13,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-25 06:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-25 06:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-25 06:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-25 06:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-25 06:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-25 06:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-25 06:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-25 06:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-25 06:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-25 06:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-25 06:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-25 06:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-25 06:53:13,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-25 06:53:13,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-25 06:53:13,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-25 06:53:13,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-25 06:53:13,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-25 06:53:13,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-25 06:53:13,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-25 06:53:13,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-25 06:53:13,793 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03985289072782563, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055672603969745896, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03985289072782563, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17483333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:53:13,793 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-25 06:53:13,794 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17975510320340182, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03985289072782563, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055672603969745896, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03985289072782563, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17483333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-25 06:53:14,244 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 577 ms to try 5000 different iterations. Found a solution that moves 2665 regions; Going from a computed imbalance of 0.17975510320340182 to a new imbalance of 0.016026918241472194. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021757199558651654); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03635156926603382, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2961111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021757199558651654); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0465, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:53:14,257 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1900443636:79 , srv1601193246:80 , srv362305706:80 , srv382506991:80 , srv630097385:80 , srv1806473470:81 , srv320083917:81 , srv74843367:81 , srv755540294:81 , srv836824847:81 , srv1709191069:82 , srv256278689:82 , srv1169040700:83 , srv1314112844:83 , srv1452460136:83 , srv1498240996:83 , srv1599818161:83 , srv1827861641:83 , srv1879025180:83 , srv1961671275:83 , srv2003029351:83 , srv280202567:83 , srv329926373:83 , srv483409803:83 , srv801749386:83 , srv1037271651:84 , srv1123067877:84 , srv1384533330:84 , srv1704878715:84 , srv1820257232:84 , srv1939724504:84 , srv2049213004:84 , srv209971219:84 , srv289357427:84 , srv313494633:84 , srv584403141:84 , srv1026723145:85 , srv1308722936:85 , srv1508967154:85 , srv2095834026:85 , srv285223653:85 , srv420524652:85 , srv586563889:85 , srv63959350:85 , srv655173373:85 , srv787688397:85 , srv8073446:85 , srv863026483:85 , srv918188240:85 , srv112544954:86 , srv1183965618:86 , srv1248211783:86 , srv1332721801:86 , srv1430939042:86 , srv1534531889:86 , srv1551471446:86 , srv1618582735:86 , srv165813791:86 , srv1877769491:86 , srv264381210:86 , srv49011267:86 , srv547663075:86 , srv619592279:86 , srv1307313360:87 , srv1323507687:87 , srv1372120207:87 , srv2043205021:87 , srv316157019:87 , srv716082739:87 , srv907638820:87 , srv1165655453:88 , srv1170367310:88 , srv12583769:88 , srv1276325417:88 , srv1428921505:88 , srv1683727846:88 , srv1733283537:88 , srv1801897985:88 , srv1944567722:88 , srv2049924525:88 , srv308878040:88 , srv571968796:88 , srv938812458:88 , srv1121661505:89 , srv356968522:89 , srv849916544:89 , srv1114077111:90 , srv1630806486:90 , srv213171444:90 , srv268381164:90 , srv688176378:90 , srv755183771:90 , srv864109712:90 , srv94521834:90 , srv974181636:90 , srv1293268284:91 , srv1209252603:92 , srv1728218454:92 , srv830823169:92 , srv594011568:523 } 2023-07-25 06:53:14,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-25 06:53:14,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1801897985=39, srv1123067877=4, srv571968796=74, srv630097385=79, srv1037271651=1, srv547663075=73, srv12583769=12, srv1323507687=18, srv836824847=91, srv716082739=83, srv1307313360=15, srv316157019=64, srv1169040700=7, srv1733283537=38, srv2003029351=49, srv1630806486=32, srv586563889=76, srv907638820=95, srv2095834026=53, srv1683727846=34, srv594011568=77, srv1900443636=45, srv1428921505=22, srv285223653=60, srv8073446=89, srv356968522=67, srv209971219=54, srv1209252603=10, srv1599818161=29, srv2043205021=50, srv313494633=63, srv1248211783=11, srv1618582735=31, srv1827861641=42, srv1877769491=43, srv256278689=56, srv420524652=70, srv382506991=69, srv213171444=55, srv1114077111=2, srv1165655453=6, srv918188240=96, srv1961671275=48, srv1170367310=8, srv1452460136=24, srv1939724504=46, srv329926373=66, srv1430939042=23, srv264381210=57, srv308878040=62, srv112544954=5, srv49011267=72, srv1276325417=13, srv1498240996=25, srv1508967154=26, srv362305706=68, srv974181636=99, srv1709191069=36, srv1534531889=27, srv1704878715=35, srv1944567722=47, srv165813791=33, srv74843367=84, srv1293268284=14, srv1879025180=44, srv94521834=98, srv1332721801=19, srv755540294=86, srv864109712=94, srv2049213004=51, srv320083917=65, srv483409803=71, srv787688397=87, srv1820257232=41, srv830823169=90, srv1384533330=21, srv1372120207=20, srv280202567=59, srv801749386=88, srv1183965618=9, srv268381164=58, srv289357427=61, srv863026483=93, srv655173373=81, srv1026723145=0, srv584403141=75, srv755183771=85, srv1601193246=30, srv1728218454=37, srv63959350=80, srv688176378=82, srv938812458=97, srv1806473470=40, srv1121661505=3, srv1314112844=17, srv1551471446=28, srv849916544=92, srv2049924525=52, srv619592279=78, srv1308722936=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-25 06:53:14,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-25 06:53:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-25 06:53:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-25 06:53:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-25 06:53:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-25 06:53:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-25 06:53:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-25 06:53:14,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-25 06:53:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-25 06:53:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-25 06:53:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-25 06:53:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-25 06:53:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-25 06:53:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-25 06:53:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-25 06:53:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-25 06:53:14,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-25 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-25 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-25 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-25 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-25 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-25 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-25 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-25 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-25 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-25 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-25 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-25 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-25 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-25 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-25 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-25 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-25 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-25 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-25 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-25 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-25 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-25 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-25 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-25 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-25 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-25 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-25 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-25 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-25 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-25 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-25 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-25 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-25 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-25 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-25 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-25 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-25 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-25 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-25 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-25 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-25 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-25 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-25 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-25 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-25 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-25 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-25 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-25 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-25 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-25 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-25 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-25 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-25 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-25 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-25 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-25 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-25 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-25 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-25 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-25 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-25 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-25 06:53:14,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-25 06:53:14,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-25 06:53:14,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-25 06:53:14,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-25 06:53:14,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-25 06:53:14,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-25 06:53:14,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-25 06:53:14,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-25 06:53:14,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-25 06:53:14,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-25 06:53:14,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-25 06:53:14,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-25 06:53:14,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-25 06:53:14,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-25 06:53:14,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-25 06:53:14,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-25 06:53:14,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-25 06:53:14,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-25 06:53:14,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-25 06:53:14,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-25 06:53:14,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-25 06:53:14,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-25 06:53:14,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-25 06:53:14,439 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021757199558651654); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03635156926603382, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021757199558651654); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0465, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:53:14,440 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-25 06:53:14,440 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016008257512628438, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021757199558651654); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03635156926603382, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021757199558651654); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0465, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-25 06:53:14,768 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 456 ms to try 5000 different iterations. Found a solution that moves 1010 regions; Going from a computed imbalance of 0.016008257512628438 to a new imbalance of 9.263410170451666E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006001205349269034); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011131294799156291); MoveCostFunction : (multiplier=7.0, imbalance=0.11222222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006001205349269034); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:53:14,770 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1209252603:80 , srv112544954:81 , srv938812458:81 , srv1728218454:82 , srv268381164:82 , srv320083917:82 , srv1307313360:83 , srv1618582735:83 , srv1630806486:83 , srv165813791:83 , srv1683727846:83 , srv1801897985:83 , srv1900443636:83 , srv280202567:83 , srv547663075:83 , srv716082739:83 , srv755183771:83 , srv1170367310:84 , srv1248211783:84 , srv12583769:84 , srv1293268284:84 , srv1372120207:84 , srv1430939042:84 , srv2003029351:84 , srv285223653:84 , srv308878040:84 , srv356968522:84 , srv688176378:84 , srv801749386:84 , srv907638820:84 , srv974181636:84 , srv1114077111:85 , srv1276325417:85 , srv1428921505:85 , srv1452460136:85 , srv1534531889:85 , srv1704878715:85 , srv1733283537:85 , srv1820257232:85 , srv209971219:85 , srv213171444:85 , srv264381210:85 , srv289357427:85 , srv316157019:85 , srv329926373:85 , srv382506991:85 , srv584403141:85 , srv755540294:85 , srv830823169:85 , srv94521834:85 , srv1026723145:86 , srv1037271651:86 , srv1121661505:86 , srv1165655453:86 , srv1183965618:86 , srv1308722936:86 , srv1314112844:86 , srv1323507687:86 , srv1508967154:86 , srv1827861641:86 , srv1879025180:86 , srv2043205021:86 , srv2049213004:86 , srv2095834026:86 , srv362305706:86 , srv483409803:86 , srv787688397:86 , srv8073446:86 , srv836824847:86 , srv863026483:86 , srv864109712:86 , srv1123067877:87 , srv1332721801:87 , srv1384533330:87 , srv1709191069:87 , srv1944567722:87 , srv2049924525:87 , srv256278689:87 , srv313494633:87 , srv420524652:87 , srv571968796:87 , srv586563889:87 , srv619592279:87 , srv63959350:87 , srv655173373:87 , srv849916544:87 , srv1169040700:88 , srv1551471446:88 , srv1599818161:88 , srv1877769491:88 , srv1961671275:88 , srv49011267:88 , srv74843367:88 , srv918188240:88 , srv1939724504:89 , srv630097385:89 , srv1498240996:90 , srv1806473470:90 , srv1601193246:91 , srv594011568:544 } 2023-07-25 06:53:14,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-25 06:53:14,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1801897985=39, srv1123067877=4, srv571968796=74, srv630097385=79, srv1037271651=1, srv547663075=73, srv12583769=12, srv1323507687=18, srv836824847=91, srv716082739=83, srv1307313360=15, srv316157019=64, srv1169040700=7, srv1733283537=38, srv2003029351=49, srv1630806486=32, srv586563889=76, srv907638820=95, srv2095834026=53, srv1683727846=34, srv594011568=77, srv1900443636=45, srv1428921505=22, srv285223653=60, srv8073446=89, srv356968522=67, srv209971219=54, srv1209252603=10, srv1599818161=29, srv2043205021=50, srv313494633=63, srv1248211783=11, srv1618582735=31, srv1827861641=42, srv1877769491=43, srv256278689=56, srv420524652=70, srv382506991=69, srv213171444=55, srv1114077111=2, srv1165655453=6, srv918188240=96, srv1961671275=48, srv1170367310=8, srv1452460136=24, srv1939724504=46, srv329926373=66, srv1430939042=23, srv264381210=57, srv308878040=62, srv112544954=5, srv49011267=72, srv1276325417=13, srv1498240996=25, srv1508967154=26, srv362305706=68, srv974181636=99, srv1709191069=36, srv1534531889=27, srv1704878715=35, srv1944567722=47, srv165813791=33, srv74843367=84, srv1293268284=14, srv1879025180=44, srv94521834=98, srv1332721801=19, srv755540294=86, srv864109712=94, srv2049213004=51, srv320083917=65, srv483409803=71, srv787688397=87, srv1820257232=41, srv830823169=90, srv1384533330=21, srv1372120207=20, srv280202567=59, srv801749386=88, srv1183965618=9, srv268381164=58, srv289357427=61, srv863026483=93, srv655173373=81, srv1026723145=0, srv584403141=75, srv755183771=85, srv1601193246=30, srv1728218454=37, srv63959350=80, srv688176378=82, srv938812458=97, srv1806473470=40, srv1121661505=3, srv1314112844=17, srv1551471446=28, srv849916544=92, srv2049924525=52, srv619592279=78, srv1308722936=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-25 06:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-25 06:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-25 06:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-25 06:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-25 06:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-25 06:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-25 06:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-25 06:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-25 06:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-25 06:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-25 06:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-25 06:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-25 06:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-25 06:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-25 06:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-25 06:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-25 06:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-25 06:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-25 06:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-25 06:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-25 06:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-25 06:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-25 06:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-25 06:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-25 06:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-25 06:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-25 06:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-25 06:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-25 06:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-25 06:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-25 06:53:14,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-25 06:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-25 06:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-25 06:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-25 06:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-25 06:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-25 06:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-25 06:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-25 06:53:14,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-25 06:53:14,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-25 06:53:14,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-25 06:53:14,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-25 06:53:14,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-25 06:53:14,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-25 06:53:14,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-25 06:53:14,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-25 06:53:14,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-25 06:53:14,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-25 06:53:14,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-25 06:53:14,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-25 06:53:14,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-25 06:53:14,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-25 06:53:14,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-25 06:53:14,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-25 06:53:14,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-25 06:53:14,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-25 06:53:14,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-25 06:53:14,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-25 06:53:14,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-25 06:53:14,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-25 06:53:14,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-25 06:53:14,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-25 06:53:14,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-25 06:53:14,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-25 06:53:14,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-25 06:53:14,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-25 06:53:14,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-25 06:53:14,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-25 06:53:14,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-25 06:53:14,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-25 06:53:14,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-25 06:53:14,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-25 06:53:14,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-25 06:53:14,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-25 06:53:14,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-25 06:53:14,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-25 06:53:14,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-25 06:53:14,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-25 06:53:14,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-25 06:53:14,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-25 06:53:14,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-25 06:53:14,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-25 06:53:14,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-25 06:53:14,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-25 06:53:14,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-25 06:53:14,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-25 06:53:14,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-25 06:53:14,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-25 06:53:14,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-25 06:53:14,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-25 06:53:14,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-25 06:53:14,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-25 06:53:14,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-25 06:53:14,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-25 06:53:14,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-25 06:53:14,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-25 06:53:14,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-25 06:53:14,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-25 06:53:14,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-25 06:53:14,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-25 06:53:14,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-25 06:53:14,939 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006001205349269034); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011131294799156291); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006001205349269034); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:53:14,939 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-25 06:53:14,940 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.192688458886215E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006001205349269034); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011131294799156291); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006001205349269034); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-25 06:53:15,291 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 457 ms to try 5000 different iterations. Found a solution that moves 258 regions; Going from a computed imbalance of 9.192688458886215E-4 to a new imbalance of 2.825866010615315E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.028666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:53:15,293 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1498240996:82 , srv1599818161:82 , srv1601193246:83 , srv1827861641:83 , srv755183771:83 , srv801749386:83 , srv1026723145:84 , srv1121661505:84 , srv1123067877:84 , srv1169040700:84 , srv1170367310:84 , srv1183965618:84 , srv1307313360:84 , srv1332721801:84 , srv1384533330:84 , srv1428921505:84 , srv1508967154:84 , srv1704878715:84 , srv1733283537:84 , srv1806473470:84 , srv1879025180:84 , srv1939724504:84 , srv2003029351:84 , srv2049924525:84 , srv2095834026:84 , srv213171444:84 , srv256278689:84 , srv289357427:84 , srv308878040:84 , srv313494633:84 , srv356968522:84 , srv382506991:84 , srv420524652:84 , srv483409803:84 , srv547663075:84 , srv571968796:84 , srv584403141:84 , srv586563889:84 , srv619592279:84 , srv630097385:84 , srv688176378:84 , srv74843367:84 , srv755540294:84 , srv830823169:84 , srv864109712:84 , srv94521834:84 , srv974181636:84 , srv1114077111:85 , srv112544954:85 , srv1165655453:85 , srv1209252603:85 , srv1248211783:85 , srv12583769:85 , srv1276325417:85 , srv1293268284:85 , srv1323507687:85 , srv1372120207:85 , srv1430939042:85 , srv1452460136:85 , srv1534531889:85 , srv1618582735:85 , srv165813791:85 , srv1683727846:85 , srv1709191069:85 , srv1728218454:85 , srv1801897985:85 , srv1820257232:85 , srv1877769491:85 , srv1900443636:85 , srv1944567722:85 , srv1961671275:85 , srv2043205021:85 , srv2049213004:85 , srv209971219:85 , srv280202567:85 , srv285223653:85 , srv316157019:85 , srv320083917:85 , srv329926373:85 , srv362305706:85 , srv49011267:85 , srv63959350:85 , srv716082739:85 , srv787688397:85 , srv8073446:85 , srv849916544:85 , srv863026483:85 , srv907638820:85 , srv918188240:85 , srv1037271651:86 , srv1308722936:86 , srv1314112844:86 , srv1551471446:86 , srv1630806486:86 , srv264381210:86 , srv268381164:86 , srv655173373:86 , srv836824847:86 , srv938812458:86 , srv594011568:630 } 2023-07-25 06:53:15,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-25 06:53:15,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1801897985=39, srv1123067877=4, srv571968796=74, srv630097385=79, srv1037271651=1, srv547663075=73, srv12583769=12, srv1323507687=18, srv836824847=91, srv716082739=83, srv1307313360=15, srv316157019=64, srv1169040700=7, srv1733283537=38, srv2003029351=49, srv1630806486=32, srv586563889=76, srv907638820=95, srv2095834026=53, srv1683727846=34, srv594011568=77, srv1900443636=45, srv1428921505=22, srv285223653=60, srv8073446=89, srv356968522=67, srv209971219=54, srv1209252603=10, srv1599818161=29, srv2043205021=50, srv313494633=63, srv1248211783=11, srv1618582735=31, srv1827861641=42, srv1877769491=43, srv256278689=56, srv420524652=70, srv382506991=69, srv213171444=55, srv1114077111=2, srv1165655453=6, srv918188240=96, srv1961671275=48, srv1170367310=8, srv1452460136=24, srv1939724504=46, srv329926373=66, srv1430939042=23, srv264381210=57, srv308878040=62, srv112544954=5, srv49011267=72, srv1276325417=13, srv1498240996=25, srv1508967154=26, srv362305706=68, srv974181636=99, srv1709191069=36, srv1534531889=27, srv1704878715=35, srv1944567722=47, srv165813791=33, srv74843367=84, srv1293268284=14, srv1879025180=44, srv94521834=98, srv1332721801=19, srv755540294=86, srv864109712=94, srv2049213004=51, srv320083917=65, srv483409803=71, srv787688397=87, srv1820257232=41, srv830823169=90, srv1384533330=21, srv1372120207=20, srv280202567=59, srv801749386=88, srv1183965618=9, srv268381164=58, srv289357427=61, srv863026483=93, srv655173373=81, srv1026723145=0, srv584403141=75, srv755183771=85, srv1601193246=30, srv1728218454=37, srv63959350=80, srv688176378=82, srv938812458=97, srv1806473470=40, srv1121661505=3, srv1314112844=17, srv1551471446=28, srv849916544=92, srv2049924525=52, srv619592279=78, srv1308722936=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-25 06:53:15,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-25 06:53:15,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-25 06:53:15,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-25 06:53:15,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-25 06:53:15,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-25 06:53:15,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-25 06:53:15,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-25 06:53:15,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-25 06:53:15,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-25 06:53:15,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-25 06:53:15,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-25 06:53:15,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-25 06:53:15,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-25 06:53:15,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-25 06:53:15,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-25 06:53:15,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-25 06:53:15,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-25 06:53:15,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-25 06:53:15,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-25 06:53:15,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-25 06:53:15,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-25 06:53:15,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-25 06:53:15,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-25 06:53:15,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-25 06:53:15,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-25 06:53:15,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-25 06:53:15,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-25 06:53:15,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-25 06:53:15,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-25 06:53:15,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-25 06:53:15,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-25 06:53:15,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-25 06:53:15,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-25 06:53:15,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-25 06:53:15,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-25 06:53:15,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-25 06:53:15,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-25 06:53:15,433 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:53:15,433 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-25 06:53:15,433 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6452105493887186E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-25 06:53:15,771 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 443 ms to try 5000 different iterations. Found a solution that moves 23 regions; Going from a computed imbalance of 2.6452105493887186E-5 to a new imbalance of 8.420813428170152E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0025555555555555557); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:53:15,772 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv12583769:83 , srv1308722936:83 , srv1314112844:83 , srv1323507687:83 , srv1709191069:83 , srv836824847:83 , srv863026483:83 , srv918188240:83 , srv1026723145:84 , srv1037271651:84 , srv1114077111:84 , srv1121661505:84 , srv1123067877:84 , srv112544954:84 , srv1165655453:84 , srv1169040700:84 , srv1170367310:84 , srv1183965618:84 , srv1209252603:84 , srv1248211783:84 , srv1276325417:84 , srv1293268284:84 , srv1332721801:84 , srv1372120207:84 , srv1384533330:84 , srv1428921505:84 , srv1430939042:84 , srv1452460136:84 , srv1508967154:84 , srv1534531889:84 , srv1551471446:84 , srv1601193246:84 , srv1618582735:84 , srv1630806486:84 , srv165813791:84 , srv1683727846:84 , srv1704878715:84 , srv1733283537:84 , srv1801897985:84 , srv1806473470:84 , srv1820257232:84 , srv1827861641:84 , srv1877769491:84 , srv1879025180:84 , srv1900443636:84 , srv1939724504:84 , srv1944567722:84 , srv1961671275:84 , srv2003029351:84 , srv2043205021:84 , srv2049213004:84 , srv2049924525:84 , srv2095834026:84 , srv209971219:84 , srv213171444:84 , srv256278689:84 , srv264381210:84 , srv268381164:84 , srv280202567:84 , srv285223653:84 , srv289357427:84 , srv308878040:84 , srv313494633:84 , srv316157019:84 , srv320083917:84 , srv329926373:84 , srv356968522:84 , srv362305706:84 , srv382506991:84 , srv420524652:84 , srv483409803:84 , srv49011267:84 , srv571968796:84 , srv584403141:84 , srv586563889:84 , srv619592279:84 , srv63959350:84 , srv655173373:84 , srv688176378:84 , srv716082739:84 , srv74843367:84 , srv755540294:84 , srv787688397:84 , srv801749386:84 , srv8073446:84 , srv830823169:84 , srv849916544:84 , srv864109712:84 , srv907638820:84 , srv938812458:84 , srv94521834:84 , srv974181636:84 , srv1307313360:85 , srv1498240996:85 , srv1599818161:85 , srv1728218454:85 , srv547663075:85 , srv630097385:85 , srv755183771:85 , srv594011568:685 } 2023-07-25 06:53:15,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-25 06:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1801897985=39, srv1123067877=4, srv571968796=74, srv630097385=79, srv1037271651=1, srv547663075=73, srv12583769=12, srv1323507687=18, srv836824847=91, srv716082739=83, srv1307313360=15, srv316157019=64, srv1169040700=7, srv1733283537=38, srv2003029351=49, srv1630806486=32, srv586563889=76, srv907638820=95, srv2095834026=53, srv1683727846=34, srv594011568=77, srv1900443636=45, srv1428921505=22, srv285223653=60, srv8073446=89, srv356968522=67, srv209971219=54, srv1209252603=10, srv1599818161=29, srv2043205021=50, srv313494633=63, srv1248211783=11, srv1618582735=31, srv1827861641=42, srv1877769491=43, srv256278689=56, srv420524652=70, srv382506991=69, srv213171444=55, srv1114077111=2, srv1165655453=6, srv918188240=96, srv1961671275=48, srv1170367310=8, srv1452460136=24, srv1939724504=46, srv329926373=66, srv1430939042=23, srv264381210=57, srv308878040=62, srv112544954=5, srv49011267=72, srv1276325417=13, srv1498240996=25, srv1508967154=26, srv362305706=68, srv974181636=99, srv1709191069=36, srv1534531889=27, srv1704878715=35, srv1944567722=47, srv165813791=33, srv74843367=84, srv1293268284=14, srv1879025180=44, srv94521834=98, srv1332721801=19, srv755540294=86, srv864109712=94, srv2049213004=51, srv320083917=65, srv483409803=71, srv787688397=87, srv1820257232=41, srv830823169=90, srv1384533330=21, srv1372120207=20, srv280202567=59, srv801749386=88, srv1183965618=9, srv268381164=58, srv289357427=61, srv863026483=93, srv655173373=81, srv1026723145=0, srv584403141=75, srv755183771=85, srv1601193246=30, srv1728218454=37, srv63959350=80, srv688176378=82, srv938812458=97, srv1806473470=40, srv1121661505=3, srv1314112844=17, srv1551471446=28, srv849916544=92, srv2049924525=52, srv619592279=78, srv1308722936=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-25 06:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-25 06:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-25 06:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-25 06:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-25 06:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-25 06:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-25 06:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-25 06:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-25 06:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-25 06:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-25 06:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-25 06:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-25 06:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-25 06:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-25 06:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-25 06:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-25 06:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-25 06:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-25 06:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-25 06:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-25 06:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-25 06:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-25 06:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-25 06:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-25 06:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-25 06:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-25 06:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-25 06:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-25 06:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-25 06:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-25 06:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-25 06:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-25 06:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-25 06:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-25 06:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-25 06:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-25 06:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-25 06:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-25 06:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-25 06:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-25 06:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-25 06:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-25 06:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-25 06:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-25 06:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-25 06:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-25 06:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-25 06:53:15,948 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.259763985991403E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:53:15,948 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv12583769:83 , srv1308722936:83 , srv1314112844:83 , srv1323507687:83 , srv1709191069:83 , srv836824847:83 , srv863026483:83 , srv918188240:83 , srv1026723145:84 , srv1037271651:84 , srv1114077111:84 , srv1121661505:84 , srv1123067877:84 , srv112544954:84 , srv1165655453:84 , srv1169040700:84 , srv1170367310:84 , srv1183965618:84 , srv1209252603:84 , srv1248211783:84 , srv1276325417:84 , srv1293268284:84 , srv1332721801:84 , srv1372120207:84 , srv1384533330:84 , srv1428921505:84 , srv1430939042:84 , srv1452460136:84 , srv1508967154:84 , srv1534531889:84 , srv1551471446:84 , srv1601193246:84 , srv1618582735:84 , srv1630806486:84 , srv165813791:84 , srv1683727846:84 , srv1704878715:84 , srv1733283537:84 , srv1801897985:84 , srv1806473470:84 , srv1820257232:84 , srv1827861641:84 , srv1877769491:84 , srv1879025180:84 , srv1900443636:84 , srv1939724504:84 , srv1944567722:84 , srv1961671275:84 , srv2003029351:84 , srv2043205021:84 , srv2049213004:84 , srv2049924525:84 , srv2095834026:84 , srv209971219:84 , srv213171444:84 , srv256278689:84 , srv264381210:84 , srv268381164:84 , srv280202567:84 , srv285223653:84 , srv289357427:84 , srv308878040:84 , srv313494633:84 , srv316157019:84 , srv320083917:84 , srv329926373:84 , srv356968522:84 , srv362305706:84 , srv382506991:84 , srv420524652:84 , srv483409803:84 , srv49011267:84 , srv571968796:84 , srv584403141:84 , srv586563889:84 , srv619592279:84 , srv63959350:84 , srv655173373:84 , srv688176378:84 , srv716082739:84 , srv74843367:84 , srv755540294:84 , srv787688397:84 , srv801749386:84 , srv8073446:84 , srv830823169:84 , srv849916544:84 , srv864109712:84 , srv907638820:84 , srv938812458:84 , srv94521834:84 , srv974181636:84 , srv1307313360:85 , srv1498240996:85 , srv1599818161:85 , srv1728218454:85 , srv547663075:85 , srv630097385:85 , srv755183771:85 , srv594011568:685 } 2023-07-25 06:53:15,993 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=334 (was 311) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 173), AvailableMemoryMB=9626 (was 10389) 2023-07-25 06:53:16,004 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=334, ProcessCount=173, AvailableMemoryMB=9625 2023-07-25 06:53:16,005 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-25 06:53:16,006 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-07-25 06:53:16,021 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1007708576:84 , srv104423192:84 , srv1051269960:84 , srv1108379700:84 , srv116670451:84 , srv1213433837:84 , srv1387852236:84 , srv150509364:84 , srv1539005852:84 , srv1539170371:84 , srv1604567868:84 , srv16723137:84 , srv1765633062:84 , srv1801867:84 , srv1810360479:84 , srv1850521962:84 , srv1859420953:84 , srv1877574603:84 , srv188192305:84 , srv2088435823:84 , srv403241010:84 , srv4092242:84 , srv568352409:84 , srv578279855:84 , srv590455028:84 , srv658769665:84 , srv693984423:84 , srv741289594:84 , srv817906671:84 , srv1805768371:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-07-25 06:53:16,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-25 06:53:16,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1850521962=16, srv1810360479=15, srv568352409=23, srv1604567868=10, srv188192305=19, srv2088435823=20, srv1051269960=2, srv4092242=22, srv578279855=24, srv741289594=28, srv1007708576=0, srv590455028=25, srv1539005852=8, srv16723137=11, srv403241010=21, srv1801867=13, srv116670451=4, srv1387852236=6, srv1765633062=12, srv1108379700=3, srv1213433837=5, srv658769665=26, srv1805768371=14, srv1539170371=9, srv817906671=29, srv1877574603=18, srv693984423=27, srv150509364=7, srv104423192=1, srv1859420953=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-25 06:53:16,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-25 06:53:16,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-25 06:53:16,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-25 06:53:16,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-25 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-25 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-25 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-25 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-25 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-25 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-25 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-25 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-25 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-25 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-25 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-25 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-25 06:53:16,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-25 06:53:16,062 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0871279812861743, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:53:16,063 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912467580688322, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0871279812861743, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-07-25 06:55:16,043 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5652476 different iterations. Found a solution that moves 1841 regions; Going from a computed imbalance of 0.9912467580688322 to a new imbalance of 4.296985841319951E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6818518518518518, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:55:16,051 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1007708576:90 , srv104423192:90 , srv1051269960:90 , srv1108379700:90 , srv116670451:90 , srv1213433837:90 , srv1387852236:90 , srv150509364:90 , srv1539005852:90 , srv1539170371:90 , srv1604567868:90 , srv16723137:90 , srv1765633062:90 , srv1801867:90 , srv1805768371:90 , srv1810360479:90 , srv1850521962:90 , srv1859420953:90 , srv1877574603:90 , srv188192305:90 , srv2088435823:90 , srv403241010:90 , srv4092242:90 , srv568352409:90 , srv578279855:90 , srv590455028:90 , srv658769665:90 , srv693984423:90 , srv741289594:90 , srv817906671:90 } 2023-07-25 06:55:16,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-25 06:55:16,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1850521962=16, srv1810360479=15, srv568352409=23, srv1604567868=10, srv188192305=19, srv2088435823=20, srv1051269960=2, srv4092242=22, srv578279855=24, srv741289594=28, srv1007708576=0, srv590455028=25, srv1539005852=8, srv16723137=11, srv403241010=21, srv1801867=13, srv116670451=4, srv1387852236=6, srv1765633062=12, srv1108379700=3, srv1213433837=5, srv658769665=26, srv1805768371=14, srv1539170371=9, srv817906671=29, srv1877574603=18, srv693984423=27, srv150509364=7, srv104423192=1, srv1859420953=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-25 06:55:16,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-25 06:55:16,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-25 06:55:16,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-25 06:55:16,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-25 06:55:16,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-25 06:55:16,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-25 06:55:16,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-25 06:55:16,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-25 06:55:16,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-25 06:55:16,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-25 06:55:16,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-25 06:55:16,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-25 06:55:16,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-25 06:55:16,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-25 06:55:16,069 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-25 06:55:16,069 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1007708576:90 , srv104423192:90 , srv1051269960:90 , srv1108379700:90 , srv116670451:90 , srv1213433837:90 , srv1387852236:90 , srv150509364:90 , srv1539005852:90 , srv1539170371:90 , srv1604567868:90 , srv16723137:90 , srv1765633062:90 , srv1801867:90 , srv1805768371:90 , srv1810360479:90 , srv1850521962:90 , srv1859420953:90 , srv1877574603:90 , srv188192305:90 , srv2088435823:90 , srv403241010:90 , srv4092242:90 , srv568352409:90 , srv578279855:90 , srv590455028:90 , srv658769665:90 , srv693984423:90 , srv741289594:90 , srv817906671:90 } 2023-07-25 06:55:16,082 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=206 (was 334), ProcessCount=170 (was 173), AvailableMemoryMB=9648 (was 9625) - AvailableMemoryMB LEAK? -