2023-05-26 01:53:25,165 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-26 01:53:25,436 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-26 01:53:25,538 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-26 01:53:25,594 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-26 01:53:25,601 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-26 01:53:25,614 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=303, ProcessCount=174, AvailableMemoryMB=7563 2023-05-26 01:53:25,616 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-26 01:53:25,616 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-26 01:53:25,749 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011076733:84 , srv1034750717:84 , srv1058680451:84 , srv1059579233:84 , srv111378412:84 , srv1120534095:84 , srv1124798276:84 , srv1153871292:84 , srv1154544023:84 , srv1192152742:84 , srv1209793715:84 , srv1211102577:84 , srv1215292674:84 , srv1234183246:84 , srv126103881:84 , srv1277619622:84 , srv1280296256:84 , srv1302005021:84 , srv1320318862:84 , srv1347330141:84 , srv1349236408:84 , srv1361665631:84 , srv1376850609:84 , srv1399011072:84 , srv1402934904:84 , srv1441726856:84 , srv1471628100:84 , srv148786758:84 , srv1500718444:84 , srv1500909928:84 , srv1507231563:84 , srv1582397098:84 , srv1611196096:84 , srv1708565524:84 , srv1727897026:84 , srv1734040239:84 , srv1783689755:84 , srv1783971587:84 , srv1803496868:84 , srv1827457379:84 , srv1830532452:84 , srv184243007:84 , srv1857068175:84 , srv1874495039:84 , srv1890863559:84 , srv1906095949:84 , srv1927245710:84 , srv1950992900:84 , srv1962778370:84 , srv2013194507:84 , srv2081358371:84 , srv2095886122:84 , srv211995713:84 , srv2126237424:84 , srv2139678572:84 , srv232569466:84 , srv236882395:84 , srv242459565:84 , srv244505378:84 , srv268535256:84 , srv272401525:84 , srv31308155:84 , srv32640604:84 , srv365998849:84 , srv376082918:84 , srv404507087:84 , srv404589664:84 , srv444027991:84 , srv449389941:84 , srv452936010:84 , srv481419909:84 , srv515244988:84 , srv567134535:84 , srv579246490:84 , srv601030905:84 , srv637051670:84 , srv64344629:84 , srv665078590:84 , srv681776624:84 , srv69332271:84 , srv707315804:84 , srv71325524:84 , srv715767169:84 , srv719141303:84 , srv721773840:84 , srv734305085:84 , srv746871964:84 , srv759149717:84 , srv770033431:84 , srv777733927:84 , srv785552635:84 , srv804571711:84 , srv804694375:84 , srv860798537:84 , srv877599376:84 , srv878104772:84 , srv902662800:84 , srv903482696:84 , srv951465298:84 , srv476982541:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-26 01:53:25,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-26 01:53:25,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1927245710=46, srv681776624=79, srv1277619622=15, srv1347330141=19, srv902662800=97, srv1906095949=45, srv2139678572=54, srv1192152742=9, srv1500718444=28, srv211995713=52, srv1302005021=17, srv1399011072=23, srv1611196096=32, srv1727897026=34, srv715767169=83, srv1211102577=11, srv1962778370=48, srv785552635=91, srv1124798276=6, srv272401525=60, srv1827457379=39, srv1320318862=18, srv1830532452=40, srv244505378=58, srv481419909=71, srv232569466=55, srv1874495039=43, srv404507087=65, srv1507231563=30, srv1734040239=35, srv452936010=69, srv1120534095=5, srv111378412=4, srv1209793715=10, srv1708565524=33, srv376082918=64, srv1500909928=29, srv71325524=82, srv1153871292=7, srv1803496868=38, srv236882395=56, srv69332271=80, srv1857068175=42, srv64344629=77, srv126103881=14, srv860798537=94, srv1349236408=20, srv1059579233=3, srv903482696=98, srv1215292674=12, srv515244988=72, srv1783689755=36, srv579246490=74, srv759149717=88, srv1034750717=1, srv951465298=99, srv1361665631=21, srv1058680451=2, srv365998849=63, srv1402934904=24, srv1441726856=25, srv804571711=92, srv1376850609=22, srv1950992900=47, srv148786758=27, srv707315804=81, srv721773840=85, srv184243007=41, srv476982541=70, srv31308155=61, srv1011076733=0, srv242459565=57, srv746871964=87, srv444027991=67, srv719141303=84, srv32640604=62, srv770033431=89, srv601030905=75, srv1280296256=16, srv1154544023=8, srv567134535=73, srv2126237424=53, srv2081358371=50, srv777733927=90, srv804694375=93, srv2095886122=51, srv2013194507=49, srv665078590=78, srv404589664=66, srv1471628100=26, srv734305085=86, srv877599376=95, srv268535256=59, srv1582397098=31, srv637051670=76, srv1890863559=44, srv878104772=96, srv449389941=68, srv1234183246=13, srv1783971587=37} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-26 01:53:25,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-26 01:53:25,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-26 01:53:25,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-26 01:53:25,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-26 01:53:25,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-26 01:53:25,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-26 01:53:25,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-26 01:53:25,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-26 01:53:25,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-26 01:53:25,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-26 01:53:25,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-26 01:53:25,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-26 01:53:25,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-26 01:53:25,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-26 01:53:25,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-26 01:53:25,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-26 01:53:25,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-26 01:53:25,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-26 01:53:25,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-26 01:53:25,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-26 01:53:25,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-26 01:53:25,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-26 01:53:25,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-26 01:53:25,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-26 01:53:25,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-26 01:53:25,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-26 01:53:25,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-26 01:53:25,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-26 01:53:25,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-26 01:53:25,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-26 01:53:25,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-26 01:53:25,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-26 01:53:25,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-26 01:53:25,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-26 01:53:25,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-26 01:53:25,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-26 01:53:25,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-26 01:53:25,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-26 01:53:25,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-26 01:53:25,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-26 01:53:25,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-26 01:53:25,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-26 01:53:25,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-26 01:53:25,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-26 01:53:25,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-26 01:53:25,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-26 01:53:25,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-26 01:53:25,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-26 01:53:25,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-26 01:53:25,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-26 01:53:25,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-26 01:53:25,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-26 01:53:25,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-26 01:53:25,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-26 01:53:25,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-26 01:53:25,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-26 01:53:25,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-26 01:53:25,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-26 01:53:25,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-26 01:53:25,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-26 01:53:25,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-26 01:53:25,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-26 01:53:25,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-26 01:53:25,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-26 01:53:25,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-26 01:53:25,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-26 01:53:25,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-26 01:53:25,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-26 01:53:25,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-26 01:53:25,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-26 01:53:25,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-26 01:53:25,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-26 01:53:25,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-26 01:53:25,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-26 01:53:25,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-26 01:53:25,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-26 01:53:25,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-26 01:53:25,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-26 01:53:25,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-26 01:53:25,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-26 01:53:25,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-26 01:53:25,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-26 01:53:25,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-26 01:53:25,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-26 01:53:25,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-26 01:53:25,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-26 01:53:25,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-26 01:53:25,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-26 01:53:25,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-26 01:53:25,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-26 01:53:25,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-26 01:53:25,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-26 01:53:25,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-26 01:53:25,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-26 01:53:25,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-26 01:53:25,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-26 01:53:25,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-26 01:53:25,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-26 01:53:25,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-26 01:53:25,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-26 01:53:25,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-26 01:53:25,991 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:53:25,992 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-26 01:53:25,992 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-26 01:53:26,309 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 449 ms to try 5000 different iterations. Found a solution that moves 4680 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17839109020400307. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04285863333888828, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055488848840816764, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.52, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04285863333888828, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:53:26,319 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1783689755:80 , srv184243007:80 , srv1234183246:82 , srv1302005021:82 , srv1402934904:82 , srv2081358371:82 , srv1192152742:83 , srv1277619622:83 , srv1349236408:83 , srv1890863559:83 , srv2013194507:83 , srv365998849:83 , srv1034750717:84 , srv1215292674:84 , srv1803496868:84 , srv1827457379:84 , srv1950992900:84 , srv2095886122:84 , srv2126237424:84 , srv481419909:84 , srv601030905:84 , srv637051670:84 , srv804694375:84 , srv877599376:84 , srv902662800:84 , srv1120534095:85 , srv1347330141:85 , srv1441726856:85 , srv1582397098:85 , srv1727897026:85 , srv1874495039:85 , srv272401525:85 , srv32640604:85 , srv444027991:85 , srv452936010:85 , srv64344629:85 , srv681776624:85 , srv69332271:85 , srv707315804:85 , srv715767169:85 , srv721773840:85 , srv746871964:85 , srv1011076733:86 , srv1209793715:86 , srv1211102577:86 , srv1320318862:86 , srv1399011072:86 , srv1783971587:86 , srv1830532452:86 , srv1857068175:86 , srv211995713:86 , srv242459565:86 , srv404589664:86 , srv449389941:86 , srv515244988:86 , srv770033431:86 , srv777733927:86 , srv785552635:86 , srv903482696:86 , srv1153871292:87 , srv126103881:87 , srv1361665631:87 , srv1376850609:87 , srv1471628100:87 , srv1500718444:87 , srv1500909928:87 , srv1611196096:87 , srv2139678572:87 , srv236882395:87 , srv376082918:87 , srv404507087:87 , srv567134535:87 , srv665078590:87 , srv1154544023:88 , srv1280296256:88 , srv1507231563:88 , srv1708565524:88 , srv1927245710:88 , srv244505378:88 , srv268535256:88 , srv31308155:88 , srv1734040239:89 , srv1906095949:89 , srv232569466:89 , srv71325524:89 , srv719141303:89 , srv734305085:89 , srv759149717:89 , srv804571711:89 , srv951465298:89 , srv1059579233:90 , srv1124798276:90 , srv148786758:90 , srv1962778370:90 , srv579246490:90 , srv860798537:91 , srv878104772:91 , srv111378412:92 , srv1058680451:97 , srv476982541:471 } 2023-05-26 01:53:26,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-26 01:53:26,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1927245710=46, srv681776624=79, srv1277619622=15, srv1347330141=19, srv902662800=97, srv1906095949=45, srv2139678572=54, srv1192152742=9, srv1500718444=28, srv211995713=52, srv1302005021=17, srv1399011072=23, srv1611196096=32, srv1727897026=34, srv715767169=83, srv1211102577=11, srv1962778370=48, srv785552635=91, srv1124798276=6, srv272401525=60, srv1827457379=39, srv1320318862=18, srv1830532452=40, srv244505378=58, srv481419909=71, srv232569466=55, srv1874495039=43, srv404507087=65, srv1507231563=30, srv1734040239=35, srv452936010=69, srv1120534095=5, srv111378412=4, srv1209793715=10, srv1708565524=33, srv376082918=64, srv1500909928=29, srv71325524=82, srv1153871292=7, srv1803496868=38, srv236882395=56, srv69332271=80, srv1857068175=42, srv64344629=77, srv126103881=14, srv860798537=94, srv1349236408=20, srv1059579233=3, srv903482696=98, srv1215292674=12, srv515244988=72, srv1783689755=36, srv579246490=74, srv759149717=88, srv1034750717=1, srv951465298=99, srv1361665631=21, srv1058680451=2, srv365998849=63, srv1402934904=24, srv1441726856=25, srv804571711=92, srv1376850609=22, srv1950992900=47, srv148786758=27, srv707315804=81, srv721773840=85, srv184243007=41, srv476982541=70, srv31308155=61, srv1011076733=0, srv242459565=57, srv746871964=87, srv444027991=67, srv719141303=84, srv32640604=62, srv770033431=89, srv601030905=75, srv1280296256=16, srv1154544023=8, srv567134535=73, srv2126237424=53, srv2081358371=50, srv777733927=90, srv804694375=93, srv2095886122=51, srv2013194507=49, srv665078590=78, srv404589664=66, srv1471628100=26, srv734305085=86, srv877599376=95, srv268535256=59, srv1582397098=31, srv637051670=76, srv1890863559=44, srv878104772=96, srv449389941=68, srv1234183246=13, srv1783971587=37} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-26 01:53:26,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-26 01:53:26,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-26 01:53:26,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-26 01:53:26,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-26 01:53:26,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-26 01:53:26,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-26 01:53:26,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-26 01:53:26,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-26 01:53:26,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-26 01:53:26,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-26 01:53:26,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-26 01:53:26,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-26 01:53:26,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-26 01:53:26,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-26 01:53:26,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-26 01:53:26,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-26 01:53:26,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-26 01:53:26,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-26 01:53:26,456 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04285863333888828, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055488848840816764, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04285863333888828, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:53:26,457 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-26 01:53:26,457 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17835832014359448, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04285863333888828, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055488848840816764, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04285863333888828, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-26 01:53:26,768 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 410 ms to try 5000 different iterations. Found a solution that moves 2641 regions; Going from a computed imbalance of 0.17835832014359448 to a new imbalance of 0.01446194746510543. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02396738857648984); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03469281583063181, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29344444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02396738857648984); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:53:26,782 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1058680451:78 , srv579246490:80 , srv148786758:81 , srv734305085:81 , srv878104772:81 , srv951465298:81 , srv111378412:82 , srv1124798276:82 , srv1500909928:82 , srv1507231563:82 , srv1611196096:82 , srv1708565524:82 , srv1734040239:82 , srv232569466:82 , srv71325524:82 , srv759149717:82 , srv860798537:82 , srv1280296256:83 , srv1471628100:83 , srv1830532452:83 , srv1906095949:83 , srv376082918:83 , srv404507087:83 , srv404589664:83 , srv1059579233:84 , srv1927245710:84 , srv1962778370:84 , srv31308155:84 , srv707315804:84 , srv719141303:84 , srv770033431:84 , srv804571711:84 , srv1011076733:85 , srv1120534095:85 , srv1154544023:85 , srv126103881:85 , srv1347330141:85 , srv1399011072:85 , srv1727897026:85 , srv1857068175:85 , srv236882395:85 , srv242459565:85 , srv244505378:85 , srv268535256:85 , srv452936010:85 , srv567134535:85 , srv1209793715:86 , srv1211102577:86 , srv1320318862:86 , srv1361665631:86 , srv1783971587:86 , srv1874495039:86 , srv365998849:86 , srv449389941:86 , srv515244988:86 , srv637051670:86 , srv665078590:86 , srv721773840:86 , srv903482696:86 , srv1153871292:87 , srv1376850609:87 , srv1500718444:87 , srv1827457379:87 , srv1950992900:87 , srv2139678572:87 , srv272401525:87 , srv444027991:87 , srv64344629:87 , srv69332271:87 , srv715767169:87 , srv804694375:87 , srv1215292674:88 , srv1441726856:88 , srv1582397098:88 , srv1803496868:88 , srv2013194507:88 , srv2081358371:88 , srv2095886122:88 , srv2126237424:88 , srv32640604:88 , srv601030905:88 , srv681776624:88 , srv746871964:88 , srv902662800:88 , srv1034750717:89 , srv1234183246:89 , srv1277619622:89 , srv1349236408:89 , srv211995713:89 , srv481419909:89 , srv777733927:89 , srv785552635:89 , srv877599376:89 , srv1783689755:91 , srv184243007:91 , srv1890863559:91 , srv1192152742:92 , srv1402934904:92 , srv1302005021:93 , srv476982541:516 } 2023-05-26 01:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-26 01:53:26,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1927245710=46, srv681776624=79, srv1277619622=15, srv1347330141=19, srv902662800=97, srv1906095949=45, srv2139678572=54, srv1192152742=9, srv1500718444=28, srv211995713=52, srv1302005021=17, srv1399011072=23, srv1611196096=32, srv1727897026=34, srv715767169=83, srv1211102577=11, srv1962778370=48, srv785552635=91, srv1124798276=6, srv272401525=60, srv1827457379=39, srv1320318862=18, srv1830532452=40, srv244505378=58, srv481419909=71, srv232569466=55, srv1874495039=43, srv404507087=65, srv1507231563=30, srv1734040239=35, srv452936010=69, srv1120534095=5, srv111378412=4, srv1209793715=10, srv1708565524=33, srv376082918=64, srv1500909928=29, srv71325524=82, srv1153871292=7, srv1803496868=38, srv236882395=56, srv69332271=80, srv1857068175=42, srv64344629=77, srv126103881=14, srv860798537=94, srv1349236408=20, srv1059579233=3, srv903482696=98, srv1215292674=12, srv515244988=72, srv1783689755=36, srv579246490=74, srv759149717=88, srv1034750717=1, srv951465298=99, srv1361665631=21, srv1058680451=2, srv365998849=63, srv1402934904=24, srv1441726856=25, srv804571711=92, srv1376850609=22, srv1950992900=47, srv148786758=27, srv707315804=81, srv721773840=85, srv184243007=41, srv476982541=70, srv31308155=61, srv1011076733=0, srv242459565=57, srv746871964=87, srv444027991=67, srv719141303=84, srv32640604=62, srv770033431=89, srv601030905=75, srv1280296256=16, srv1154544023=8, srv567134535=73, srv2126237424=53, srv2081358371=50, srv777733927=90, srv804694375=93, srv2095886122=51, srv2013194507=49, srv665078590=78, srv404589664=66, srv1471628100=26, srv734305085=86, srv877599376=95, srv268535256=59, srv1582397098=31, srv637051670=76, srv1890863559=44, srv878104772=96, srv449389941=68, srv1234183246=13, srv1783971587=37} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-26 01:53:26,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-26 01:53:26,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-26 01:53:26,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-26 01:53:26,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-26 01:53:26,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-26 01:53:26,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-26 01:53:26,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-26 01:53:26,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-26 01:53:26,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-26 01:53:26,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-26 01:53:26,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-26 01:53:26,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-26 01:53:26,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-26 01:53:26,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-26 01:53:26,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-26 01:53:26,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-26 01:53:26,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-26 01:53:26,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-26 01:53:26,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-26 01:53:26,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-26 01:53:26,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-26 01:53:26,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-26 01:53:26,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-26 01:53:26,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-26 01:53:26,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-26 01:53:26,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-26 01:53:26,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-26 01:53:26,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-26 01:53:26,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-26 01:53:26,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-26 01:53:26,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-26 01:53:26,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-26 01:53:26,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-26 01:53:26,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-26 01:53:26,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-26 01:53:26,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-26 01:53:26,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-26 01:53:26,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-26 01:53:26,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-26 01:53:26,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-26 01:53:26,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-26 01:53:26,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-26 01:53:26,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-26 01:53:26,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-26 01:53:26,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-26 01:53:26,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-26 01:53:26,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-26 01:53:26,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-26 01:53:26,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-26 01:53:26,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-26 01:53:26,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-26 01:53:26,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-26 01:53:26,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-26 01:53:26,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-26 01:53:26,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-26 01:53:26,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-26 01:53:26,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-26 01:53:26,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-26 01:53:26,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-26 01:53:26,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-26 01:53:26,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-26 01:53:26,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-26 01:53:26,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-26 01:53:26,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-26 01:53:26,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-26 01:53:26,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-26 01:53:26,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-26 01:53:26,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-26 01:53:26,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-26 01:53:26,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-26 01:53:26,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-26 01:53:26,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-26 01:53:26,939 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02396738857648984); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03469281583063181, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02396738857648984); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:53:26,940 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-26 01:53:26,940 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014443454787853513, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02396738857648984); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03469281583063181, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02396738857648984); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-26 01:53:27,186 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 340 ms to try 5000 different iterations. Found a solution that moves 996 regions; Going from a computed imbalance of 0.014443454787853513 to a new imbalance of 6.203363440570493E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008260625575862978); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00908866414480969); MoveCostFunction : (multiplier=7.0, imbalance=0.11066666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008260625575862978); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:53:27,187 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv211995713:81 , srv1059579233:82 , srv1962778370:82 , srv1058680451:83 , srv1153871292:83 , srv1302005021:83 , srv1376850609:83 , srv1402934904:83 , srv1890863559:83 , srv244505378:83 , srv32640604:83 , srv719141303:83 , srv777733927:83 , srv785552635:83 , srv1034750717:84 , srv111378412:84 , srv1215292674:84 , srv1320318862:84 , srv1441726856:84 , srv1500718444:84 , srv1582397098:84 , srv2139678572:84 , srv601030905:84 , srv69332271:84 , srv804571711:84 , srv877599376:84 , srv1011076733:85 , srv1124798276:85 , srv1154544023:85 , srv1192152742:85 , srv1209793715:85 , srv1211102577:85 , srv1277619622:85 , srv1349236408:85 , srv1361665631:85 , srv1399011072:85 , srv1734040239:85 , srv1783689755:85 , srv1906095949:85 , srv2126237424:85 , srv242459565:85 , srv268535256:85 , srv272401525:85 , srv449389941:85 , srv481419909:85 , srv665078590:85 , srv681776624:85 , srv746871964:85 , srv860798537:85 , srv1120534095:86 , srv1234183246:86 , srv1280296256:86 , srv1471628100:86 , srv148786758:86 , srv1507231563:86 , srv1803496868:86 , srv1857068175:86 , srv1927245710:86 , srv1950992900:86 , srv2013194507:86 , srv2081358371:86 , srv2095886122:86 , srv232569466:86 , srv31308155:86 , srv376082918:86 , srv404507087:86 , srv444027991:86 , srv515244988:86 , srv579246490:86 , srv64344629:86 , srv71325524:86 , srv715767169:86 , srv721773840:86 , srv759149717:86 , srv804694375:86 , srv903482696:86 , srv126103881:87 , srv1611196096:87 , srv1727897026:87 , srv1783971587:87 , srv1827457379:87 , srv1874495039:87 , srv236882395:87 , srv365998849:87 , srv567134535:87 , srv707315804:87 , srv878104772:87 , srv902662800:87 , srv951465298:87 , srv1347330141:88 , srv1500909928:88 , srv1708565524:88 , srv184243007:88 , srv452936010:88 , srv637051670:88 , srv734305085:88 , srv770033431:88 , srv1830532452:89 , srv404589664:89 , srv476982541:544 } 2023-05-26 01:53:27,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-26 01:53:27,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1927245710=46, srv681776624=79, srv1277619622=15, srv1347330141=19, srv902662800=97, srv1906095949=45, srv2139678572=54, srv1192152742=9, srv1500718444=28, srv211995713=52, srv1302005021=17, srv1399011072=23, srv1611196096=32, srv1727897026=34, srv715767169=83, srv1211102577=11, srv1962778370=48, srv785552635=91, srv1124798276=6, srv272401525=60, srv1827457379=39, srv1320318862=18, srv1830532452=40, srv244505378=58, srv481419909=71, srv232569466=55, srv1874495039=43, srv404507087=65, srv1507231563=30, srv1734040239=35, srv452936010=69, srv1120534095=5, srv111378412=4, srv1209793715=10, srv1708565524=33, srv376082918=64, srv1500909928=29, srv71325524=82, srv1153871292=7, srv1803496868=38, srv236882395=56, srv69332271=80, srv1857068175=42, srv64344629=77, srv126103881=14, srv860798537=94, srv1349236408=20, srv1059579233=3, srv903482696=98, srv1215292674=12, srv515244988=72, srv1783689755=36, srv579246490=74, srv759149717=88, srv1034750717=1, srv951465298=99, srv1361665631=21, srv1058680451=2, srv365998849=63, srv1402934904=24, srv1441726856=25, srv804571711=92, srv1376850609=22, srv1950992900=47, srv148786758=27, srv707315804=81, srv721773840=85, srv184243007=41, srv476982541=70, srv31308155=61, srv1011076733=0, srv242459565=57, srv746871964=87, srv444027991=67, srv719141303=84, srv32640604=62, srv770033431=89, srv601030905=75, srv1280296256=16, srv1154544023=8, srv567134535=73, srv2126237424=53, srv2081358371=50, srv777733927=90, srv804694375=93, srv2095886122=51, srv2013194507=49, srv665078590=78, srv404589664=66, srv1471628100=26, srv734305085=86, srv877599376=95, srv268535256=59, srv1582397098=31, srv637051670=76, srv1890863559=44, srv878104772=96, srv449389941=68, srv1234183246=13, srv1783971587=37} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-26 01:53:27,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-26 01:53:27,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-26 01:53:27,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-26 01:53:27,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-26 01:53:27,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-26 01:53:27,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-26 01:53:27,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-26 01:53:27,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-26 01:53:27,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-26 01:53:27,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-26 01:53:27,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-26 01:53:27,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-26 01:53:27,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-26 01:53:27,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-26 01:53:27,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-26 01:53:27,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-26 01:53:27,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-26 01:53:27,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-26 01:53:27,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-26 01:53:27,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-26 01:53:27,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-26 01:53:27,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-26 01:53:27,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-26 01:53:27,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-26 01:53:27,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-26 01:53:27,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-26 01:53:27,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-26 01:53:27,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-26 01:53:27,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-26 01:53:27,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-26 01:53:27,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-26 01:53:27,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-26 01:53:27,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-26 01:53:27,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-26 01:53:27,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-26 01:53:27,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-26 01:53:27,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-26 01:53:27,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-26 01:53:27,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-26 01:53:27,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-26 01:53:27,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-26 01:53:27,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-26 01:53:27,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-26 01:53:27,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-26 01:53:27,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-26 01:53:27,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-26 01:53:27,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-26 01:53:27,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-26 01:53:27,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-26 01:53:27,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-26 01:53:27,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-26 01:53:27,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-26 01:53:27,319 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008260625575862978); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00908866414480969); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008260625575862978); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:53:27,319 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-26 01:53:27,319 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.133622029957436E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008260625575862978); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00908866414480969); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008260625575862978); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-26 01:53:27,684 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 462 ms to try 5000 different iterations. Found a solution that moves 308 regions; Going from a computed imbalance of 6.133622029957436E-4 to a new imbalance of 4.386731721534986E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.03422222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:53:27,685 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1192152742:82 , srv126103881:83 , srv1783971587:83 , srv184243007:83 , srv236882395:83 , srv404589664:83 , srv902662800:83 , srv903482696:83 , srv1058680451:84 , srv111378412:84 , srv1124798276:84 , srv1154544023:84 , srv1209793715:84 , srv1302005021:84 , srv1347330141:84 , srv1361665631:84 , srv1500718444:84 , srv1708565524:84 , srv1803496868:84 , srv1827457379:84 , srv1830532452:84 , srv1874495039:84 , srv1927245710:84 , srv2013194507:84 , srv2095886122:84 , srv2139678572:84 , srv232569466:84 , srv268535256:84 , srv31308155:84 , srv444027991:84 , srv481419909:84 , srv515244988:84 , srv567134535:84 , srv637051670:84 , srv64344629:84 , srv665078590:84 , srv681776624:84 , srv71325524:84 , srv715767169:84 , srv734305085:84 , srv746871964:84 , srv770033431:84 , srv777733927:84 , srv785552635:84 , srv860798537:84 , srv878104772:84 , srv1011076733:85 , srv1034750717:85 , srv1153871292:85 , srv1211102577:85 , srv1234183246:85 , srv1277619622:85 , srv1280296256:85 , srv1320318862:85 , srv1402934904:85 , srv1441726856:85 , srv148786758:85 , srv1500909928:85 , srv1582397098:85 , srv1727897026:85 , srv1857068175:85 , srv1890863559:85 , srv1906095949:85 , srv2081358371:85 , srv2126237424:85 , srv449389941:85 , srv452936010:85 , srv721773840:85 , srv759149717:85 , srv804571711:85 , srv804694375:85 , srv877599376:85 , srv951465298:85 , srv1059579233:86 , srv1120534095:86 , srv1349236408:86 , srv1376850609:86 , srv1399011072:86 , srv1471628100:86 , srv1507231563:86 , srv1783689755:86 , srv1950992900:86 , srv1962778370:86 , srv211995713:86 , srv242459565:86 , srv244505378:86 , srv272401525:86 , srv32640604:86 , srv365998849:86 , srv376082918:86 , srv404507087:86 , srv579246490:86 , srv601030905:86 , srv69332271:86 , srv707315804:86 , srv719141303:86 , srv1215292674:87 , srv1611196096:87 , srv1734040239:87 , srv476982541:611 } 2023-05-26 01:53:27,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-26 01:53:27,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1927245710=46, srv681776624=79, srv1277619622=15, srv1347330141=19, srv902662800=97, srv1906095949=45, srv2139678572=54, srv1192152742=9, srv1500718444=28, srv211995713=52, srv1302005021=17, srv1399011072=23, srv1611196096=32, srv1727897026=34, srv715767169=83, srv1211102577=11, srv1962778370=48, srv785552635=91, srv1124798276=6, srv272401525=60, srv1827457379=39, srv1320318862=18, srv1830532452=40, srv244505378=58, srv481419909=71, srv232569466=55, srv1874495039=43, srv404507087=65, srv1507231563=30, srv1734040239=35, srv452936010=69, srv1120534095=5, srv111378412=4, srv1209793715=10, srv1708565524=33, srv376082918=64, srv1500909928=29, srv71325524=82, srv1153871292=7, srv1803496868=38, srv236882395=56, srv69332271=80, srv1857068175=42, srv64344629=77, srv126103881=14, srv860798537=94, srv1349236408=20, srv1059579233=3, srv903482696=98, srv1215292674=12, srv515244988=72, srv1783689755=36, srv579246490=74, srv759149717=88, srv1034750717=1, srv951465298=99, srv1361665631=21, srv1058680451=2, srv365998849=63, srv1402934904=24, srv1441726856=25, srv804571711=92, srv1376850609=22, srv1950992900=47, srv148786758=27, srv707315804=81, srv721773840=85, srv184243007=41, srv476982541=70, srv31308155=61, srv1011076733=0, srv242459565=57, srv746871964=87, srv444027991=67, srv719141303=84, srv32640604=62, srv770033431=89, srv601030905=75, srv1280296256=16, srv1154544023=8, srv567134535=73, srv2126237424=53, srv2081358371=50, srv777733927=90, srv804694375=93, srv2095886122=51, srv2013194507=49, srv665078590=78, srv404589664=66, srv1471628100=26, srv734305085=86, srv877599376=95, srv268535256=59, srv1582397098=31, srv637051670=76, srv1890863559=44, srv878104772=96, srv449389941=68, srv1234183246=13, srv1783971587=37} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-26 01:53:27,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-26 01:53:27,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-26 01:53:27,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-26 01:53:27,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-26 01:53:27,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-26 01:53:27,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-26 01:53:27,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-26 01:53:27,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-26 01:53:27,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-26 01:53:27,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-26 01:53:27,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-26 01:53:27,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-26 01:53:27,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-26 01:53:27,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-26 01:53:27,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-26 01:53:27,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-26 01:53:27,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-26 01:53:27,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-26 01:53:27,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-26 01:53:27,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-26 01:53:27,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-26 01:53:27,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-26 01:53:27,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-26 01:53:27,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-26 01:53:27,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-26 01:53:27,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-26 01:53:27,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-26 01:53:27,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-26 01:53:27,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-26 01:53:27,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-26 01:53:27,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-26 01:53:27,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-26 01:53:27,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-26 01:53:27,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-26 01:53:27,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-26 01:53:27,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-26 01:53:27,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-26 01:53:27,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-26 01:53:27,839 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:53:27,839 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-26 01:53:27,839 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.171065512008661E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-26 01:53:28,044 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 317 ms to try 5000 different iterations. Found a solution that moves 39 regions; Going from a computed imbalance of 4.171065512008661E-5 to a new imbalance of 6.671069509007577E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.004333333333333333); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:53:28,044 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1215292674:83 , srv1349236408:83 , srv1376850609:83 , srv1611196096:83 , srv1734040239:83 , srv1950992900:83 , srv272401525:83 , srv452936010:83 , srv567134535:83 , srv878104772:83 , srv1034750717:84 , srv1058680451:84 , srv1059579233:84 , srv111378412:84 , srv1120534095:84 , srv1153871292:84 , srv1154544023:84 , srv1192152742:84 , srv1211102577:84 , srv1234183246:84 , srv126103881:84 , srv1277619622:84 , srv1280296256:84 , srv1302005021:84 , srv1347330141:84 , srv1361665631:84 , srv1399011072:84 , srv1402934904:84 , srv1441726856:84 , srv1471628100:84 , srv148786758:84 , srv1500718444:84 , srv1500909928:84 , srv1507231563:84 , srv1582397098:84 , srv1708565524:84 , srv1727897026:84 , srv1783689755:84 , srv1783971587:84 , srv1803496868:84 , srv1827457379:84 , srv1830532452:84 , srv184243007:84 , srv1857068175:84 , srv1874495039:84 , srv1890863559:84 , srv1906095949:84 , srv1927245710:84 , srv1962778370:84 , srv2095886122:84 , srv211995713:84 , srv2126237424:84 , srv2139678572:84 , srv236882395:84 , srv242459565:84 , srv244505378:84 , srv268535256:84 , srv31308155:84 , srv32640604:84 , srv365998849:84 , srv376082918:84 , srv404507087:84 , srv444027991:84 , srv449389941:84 , srv481419909:84 , srv515244988:84 , srv579246490:84 , srv601030905:84 , srv637051670:84 , srv64344629:84 , srv665078590:84 , srv681776624:84 , srv69332271:84 , srv707315804:84 , srv715767169:84 , srv719141303:84 , srv721773840:84 , srv734305085:84 , srv746871964:84 , srv759149717:84 , srv770033431:84 , srv777733927:84 , srv785552635:84 , srv804571711:84 , srv804694375:84 , srv860798537:84 , srv877599376:84 , srv902662800:84 , srv951465298:84 , srv1011076733:85 , srv1124798276:85 , srv1209793715:85 , srv1320318862:85 , srv2013194507:85 , srv2081358371:85 , srv232569466:85 , srv404589664:85 , srv71325524:85 , srv903482696:85 , srv476982541:684 } 2023-05-26 01:53:28,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-26 01:53:28,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1927245710=46, srv681776624=79, srv1277619622=15, srv1347330141=19, srv902662800=97, srv1906095949=45, srv2139678572=54, srv1192152742=9, srv1500718444=28, srv211995713=52, srv1302005021=17, srv1399011072=23, srv1611196096=32, srv1727897026=34, srv715767169=83, srv1211102577=11, srv1962778370=48, srv785552635=91, srv1124798276=6, srv272401525=60, srv1827457379=39, srv1320318862=18, srv1830532452=40, srv244505378=58, srv481419909=71, srv232569466=55, srv1874495039=43, srv404507087=65, srv1507231563=30, srv1734040239=35, srv452936010=69, srv1120534095=5, srv111378412=4, srv1209793715=10, srv1708565524=33, srv376082918=64, srv1500909928=29, srv71325524=82, srv1153871292=7, srv1803496868=38, srv236882395=56, srv69332271=80, srv1857068175=42, srv64344629=77, srv126103881=14, srv860798537=94, srv1349236408=20, srv1059579233=3, srv903482696=98, srv1215292674=12, srv515244988=72, srv1783689755=36, srv579246490=74, srv759149717=88, srv1034750717=1, srv951465298=99, srv1361665631=21, srv1058680451=2, srv365998849=63, srv1402934904=24, srv1441726856=25, srv804571711=92, srv1376850609=22, srv1950992900=47, srv148786758=27, srv707315804=81, srv721773840=85, srv184243007=41, srv476982541=70, srv31308155=61, srv1011076733=0, srv242459565=57, srv746871964=87, srv444027991=67, srv719141303=84, srv32640604=62, srv770033431=89, srv601030905=75, srv1280296256=16, srv1154544023=8, srv567134535=73, srv2126237424=53, srv2081358371=50, srv777733927=90, srv804694375=93, srv2095886122=51, srv2013194507=49, srv665078590=78, srv404589664=66, srv1471628100=26, srv734305085=86, srv877599376=95, srv268535256=59, srv1582397098=31, srv637051670=76, srv1890863559=44, srv878104772=96, srv449389941=68, srv1234183246=13, srv1783971587=37} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-26 01:53:28,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-26 01:53:28,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-26 01:53:28,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-26 01:53:28,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-26 01:53:28,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-26 01:53:28,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-26 01:53:28,170 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:53:28,170 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1215292674:83 , srv1349236408:83 , srv1376850609:83 , srv1611196096:83 , srv1734040239:83 , srv1950992900:83 , srv272401525:83 , srv452936010:83 , srv567134535:83 , srv878104772:83 , srv1034750717:84 , srv1058680451:84 , srv1059579233:84 , srv111378412:84 , srv1120534095:84 , srv1153871292:84 , srv1154544023:84 , srv1192152742:84 , srv1211102577:84 , srv1234183246:84 , srv126103881:84 , srv1277619622:84 , srv1280296256:84 , srv1302005021:84 , srv1347330141:84 , srv1361665631:84 , srv1399011072:84 , srv1402934904:84 , srv1441726856:84 , srv1471628100:84 , srv148786758:84 , srv1500718444:84 , srv1500909928:84 , srv1507231563:84 , srv1582397098:84 , srv1708565524:84 , srv1727897026:84 , srv1783689755:84 , srv1783971587:84 , srv1803496868:84 , srv1827457379:84 , srv1830532452:84 , srv184243007:84 , srv1857068175:84 , srv1874495039:84 , srv1890863559:84 , srv1906095949:84 , srv1927245710:84 , srv1962778370:84 , srv2095886122:84 , srv211995713:84 , srv2126237424:84 , srv2139678572:84 , srv236882395:84 , srv242459565:84 , srv244505378:84 , srv268535256:84 , srv31308155:84 , srv32640604:84 , srv365998849:84 , srv376082918:84 , srv404507087:84 , srv444027991:84 , srv449389941:84 , srv481419909:84 , srv515244988:84 , srv579246490:84 , srv601030905:84 , srv637051670:84 , srv64344629:84 , srv665078590:84 , srv681776624:84 , srv69332271:84 , srv707315804:84 , srv715767169:84 , srv719141303:84 , srv721773840:84 , srv734305085:84 , srv746871964:84 , srv759149717:84 , srv770033431:84 , srv777733927:84 , srv785552635:84 , srv804571711:84 , srv804694375:84 , srv860798537:84 , srv877599376:84 , srv902662800:84 , srv951465298:84 , srv1011076733:85 , srv1124798276:85 , srv1209793715:85 , srv1320318862:85 , srv2013194507:85 , srv2081358371:85 , srv232569466:85 , srv404589664:85 , srv71325524:85 , srv903482696:85 , srv476982541:684 } 2023-05-26 01:53:28,196 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=367 (was 303) - SystemLoadAverage LEAK? -, ProcessCount=174 (was 174), AvailableMemoryMB=7061 (was 7563) 2023-05-26 01:53:28,204 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=367, ProcessCount=174, AvailableMemoryMB=7061 2023-05-26 01:53:28,204 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-26 01:53:28,205 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-26 01:53:28,212 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1218696856:84 , srv1234314387:84 , srv1258154289:84 , srv130686938:84 , srv1316191271:84 , srv1355549495:84 , srv1366882893:84 , srv1498525391:84 , srv1746577152:84 , srv1794915404:84 , srv1829613941:84 , srv1892796594:84 , srv1904150584:84 , srv2010478002:84 , srv2117829392:84 , srv2119686372:84 , srv291072362:84 , srv343651630:84 , srv406863283:84 , srv429329424:84 , srv49374124:84 , srv53306931:84 , srv534469057:84 , srv605096242:84 , srv612872157:84 , srv73956243:84 , srv880154819:84 , srv961417733:84 , srv97038903:84 , srv1721559002:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-26 01:53:28,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-26 01:53:28,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1904150584=13, srv1218696856=0, srv2117829392=15, srv1355549495=5, srv612872157=25, srv961417733=28, srv605096242=24, srv130686938=3, srv1829613941=11, srv1234314387=1, srv2010478002=14, srv534469057=23, srv429329424=20, srv1366882893=6, srv1794915404=10, srv73956243=26, srv1892796594=12, srv880154819=27, srv1498525391=7, srv291072362=17, srv53306931=22, srv1721559002=8, srv1258154289=2, srv1316191271=4, srv406863283=19, srv1746577152=9, srv97038903=29, srv2119686372=16, srv49374124=21, srv343651630=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-26 01:53:28,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-26 01:53:28,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-26 01:53:28,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-26 01:53:28,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-26 01:53:28,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-26 01:53:28,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-26 01:53:28,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-26 01:53:28,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-26 01:53:28,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-26 01:53:28,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-26 01:53:28,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-26 01:53:28,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-26 01:53:28,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-26 01:53:28,236 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.034195953046609, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:53:28,236 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991230079359573, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.034195953046609, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-26 01:55:28,224 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6787101 different iterations. Found a solution that moves 1839 regions; Going from a computed imbalance of 0.991230079359573 to a new imbalance of 4.292317741546655E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6811111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:55:28,229 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1218696856:90 , srv1234314387:90 , srv1258154289:90 , srv130686938:90 , srv1316191271:90 , srv1355549495:90 , srv1366882893:90 , srv1498525391:90 , srv1721559002:90 , srv1746577152:90 , srv1794915404:90 , srv1829613941:90 , srv1892796594:90 , srv1904150584:90 , srv2010478002:90 , srv2117829392:90 , srv2119686372:90 , srv291072362:90 , srv343651630:90 , srv406863283:90 , srv429329424:90 , srv49374124:90 , srv53306931:90 , srv534469057:90 , srv605096242:90 , srv612872157:90 , srv73956243:90 , srv880154819:90 , srv961417733:90 , srv97038903:90 } 2023-05-26 01:55:28,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-26 01:55:28,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1904150584=13, srv1218696856=0, srv2117829392=15, srv1355549495=5, srv612872157=25, srv961417733=28, srv605096242=24, srv130686938=3, srv1829613941=11, srv1234314387=1, srv2010478002=14, srv534469057=23, srv429329424=20, srv1366882893=6, srv1794915404=10, srv73956243=26, srv1892796594=12, srv880154819=27, srv1498525391=7, srv291072362=17, srv53306931=22, srv1721559002=8, srv1258154289=2, srv1316191271=4, srv406863283=19, srv1746577152=9, srv97038903=29, srv2119686372=16, srv49374124=21, srv343651630=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-26 01:55:28,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-26 01:55:28,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-26 01:55:28,247 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-26 01:55:28,247 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1218696856:90 , srv1234314387:90 , srv1258154289:90 , srv130686938:90 , srv1316191271:90 , srv1355549495:90 , srv1366882893:90 , srv1498525391:90 , srv1721559002:90 , srv1746577152:90 , srv1794915404:90 , srv1829613941:90 , srv1892796594:90 , srv1904150584:90 , srv2010478002:90 , srv2117829392:90 , srv2119686372:90 , srv291072362:90 , srv343651630:90 , srv406863283:90 , srv429329424:90 , srv49374124:90 , srv53306931:90 , srv534469057:90 , srv605096242:90 , srv612872157:90 , srv73956243:90 , srv880154819:90 , srv961417733:90 , srv97038903:90 } 2023-05-26 01:55:28,260 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=273 (was 367), ProcessCount=173 (was 174), AvailableMemoryMB=6733 (was 7061)