2023-07-18 06:53:11,215 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-07-18 06:53:11,534 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-18 06:53:11,649 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-18 06:53:11,709 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-18 06:53:11,720 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-18 06:53:11,735 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=308, ProcessCount=172, AvailableMemoryMB=7528 2023-07-18 06:53:11,737 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-18 06:53:11,737 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-18 06:53:11,903 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1003778807:84 , srv1013065194:84 , srv103551038:84 , srv1035986202:84 , srv1036774794:84 , srv1038904820:84 , srv1046767022:84 , srv1070222006:84 , srv1080157844:84 , srv111166700:84 , srv1115817217:84 , srv1210972891:84 , srv121627878:84 , srv1248308380:84 , srv1248604078:84 , srv1249811792:84 , srv1278851582:84 , srv1281963546:84 , srv1301105846:84 , srv1322963617:84 , srv1324973247:84 , srv1445681738:84 , srv1524440397:84 , srv1533979384:84 , srv1567586419:84 , srv1592168909:84 , srv1592921041:84 , srv1605527308:84 , srv1618978136:84 , srv1629688621:84 , srv1634611030:84 , srv1650197709:84 , srv1669887750:84 , srv1694276587:84 , srv1743752527:84 , srv1750483660:84 , srv1769328462:84 , srv1814945854:84 , srv1885742251:84 , srv1916443926:84 , srv1968665493:84 , srv1973293353:84 , srv1975424187:84 , srv1991520050:84 , srv1992679668:84 , srv2016334220:84 , srv2016564585:84 , srv2019558295:84 , srv2068663833:84 , srv2070440284:84 , srv2073936338:84 , srv2113258582:84 , srv2126645116:84 , srv2144934416:84 , srv270732371:84 , srv297322023:84 , srv307464998:84 , srv308647732:84 , srv348359454:84 , srv359763100:84 , srv374031600:84 , srv387237570:84 , srv39932257:84 , srv426339324:84 , srv504866369:84 , srv519024621:84 , srv521854817:84 , srv533344373:84 , srv538003432:84 , srv552703335:84 , srv559312461:84 , srv562786022:84 , srv567003524:84 , srv592018905:84 , srv596422624:84 , srv605475450:84 , srv648923070:84 , srv691416311:84 , srv692657735:84 , srv695182215:84 , srv696188599:84 , srv705621675:84 , srv73483525:84 , srv741623569:84 , srv781096581:84 , srv797091002:84 , srv816022746:84 , srv835341360:84 , srv83634375:84 , srv869109690:84 , srv878657741:84 , srv888461160:84 , srv893116700:84 , srv901079285:84 , srv949702897:84 , srv951887946:84 , srv967003987:84 , srv98555149:84 , srv987020808:84 , srv497271460:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-07-18 06:53:11,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 06:53:11,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1991520050=43, srv567003524=73, srv1278851582=16, srv521854817=67, srv1046767022=6, srv111166700=9, srv359763100=59, srv648923070=77, srv504866369=65, srv1301105846=18, srv1038904820=5, srv1248604078=14, srv1975424187=42, srv1669887750=32, srv2126645116=52, srv519024621=66, srv1248308380=13, srv1592168909=25, srv2070440284=49, srv1210972891=11, srv797091002=86, srv1650197709=31, srv605475450=76, srv1750483660=35, srv2016334220=45, srv307464998=56, srv1003778807=0, srv2068663833=48, srv73483525=83, srv967003987=97, srv103551038=2, srv1916443926=39, srv98555149=98, srv696188599=81, srv1324973247=20, srv781096581=85, srv1249811792=15, srv949702897=95, srv1973293353=41, srv691416311=78, srv426339324=63, srv878657741=91, srv1013065194=1, srv2016564585=46, srv1070222006=7, srv951887946=96, srv596422624=75, srv1629688621=29, srv1992679668=44, srv533344373=68, srv1634611030=30, srv1968665493=40, srv1036774794=4, srv1115817217=10, srv39932257=62, srv562786022=72, srv888461160=92, srv869109690=90, srv83634375=89, srv1322963617=19, srv1035986202=3, srv1605527308=27, srv2113258582=51, srv2019558295=47, srv2073936338=50, srv374031600=60, srv835341360=88, srv348359454=58, srv893116700=93, srv1769328462=36, srv552703335=70, srv1618978136=28, srv559312461=71, srv741623569=84, srv270732371=54, srv297322023=55, srv1694276587=33, srv695182215=80, srv538003432=69, srv592018905=74, srv1080157844=8, srv387237570=61, srv987020808=99, srv2144934416=53, srv692657735=79, srv901079285=94, srv1533979384=23, srv1567586419=24, srv1743752527=34, srv1814945854=37, srv1281963546=17, srv816022746=87, srv497271460=64, srv121627878=12, srv308647732=57, srv705621675=82, srv1524440397=22, srv1885742251=38, srv1445681738=21, srv1592921041=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 06:53:12,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 06:53:12,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 06:53:12,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 06:53:12,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 06:53:12,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 06:53:12,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 06:53:12,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 06:53:12,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 06:53:12,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 06:53:12,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 06:53:12,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 06:53:12,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 06:53:12,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 06:53:12,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 06:53:12,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 06:53:12,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 06:53:12,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 06:53:12,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 06:53:12,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 06:53:12,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 06:53:12,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 06:53:12,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 06:53:12,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 06:53:12,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 06:53:12,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 06:53:12,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 06:53:12,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 06:53:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 06:53:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 06:53:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 06:53:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 06:53:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 06:53:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 06:53:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 06:53:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 06:53:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 06:53:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 06:53:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 06:53:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 06:53:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 06:53:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 06:53:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 06:53:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 06:53:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 06:53:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 06:53:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 06:53:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 06:53:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 06:53:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 06:53:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 06:53:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 06:53:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 06:53:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 06:53:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 06:53:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 06:53:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 06:53:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 06:53:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 06:53:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 06:53:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 06:53:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 06:53:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 06:53:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 06:53:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 06:53:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 06:53:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 06:53:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 06:53:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 06:53:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 06:53:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 06:53:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 06:53:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 06:53:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 06:53:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 06:53:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 06:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 06:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 06:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 06:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 06:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 06:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 06:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 06:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 06:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 06:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 06:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 06:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 06:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 06:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 06:53:12,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 06:53:12,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 06:53:12,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 06:53:12,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 06:53:12,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 06:53:12,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 06:53:12,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 06:53:12,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 06:53:12,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 06:53:12,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 06:53:12,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 06:53:12,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 06:53:12,228 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 06:53:12,228 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-18 06:53:12,229 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-18 06:53:12,652 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 611 ms to try 5000 different iterations. Found a solution that moves 4705 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17858202667442954. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04186421556896937, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05726462095020053, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5227777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04186421556896937, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17391666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.239, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 06:53:12,674 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1035986202:81 , srv1445681738:81 , srv567003524:81 , srv888461160:81 , srv1769328462:82 , srv596422624:82 , srv1046767022:83 , srv1080157844:83 , srv1278851582:83 , srv1605527308:83 , srv1694276587:83 , srv521854817:83 , srv605475450:83 , srv111166700:84 , srv1210972891:84 , srv1322963617:84 , srv1669887750:84 , srv1814945854:84 , srv1916443926:84 , srv2016334220:84 , srv2113258582:84 , srv2144934416:84 , srv297322023:84 , srv387237570:84 , srv39932257:84 , srv504866369:84 , srv533344373:84 , srv696188599:84 , srv797091002:84 , srv1003778807:85 , srv1533979384:85 , srv1592168909:85 , srv1592921041:85 , srv1618978136:85 , srv1750483660:85 , srv1991520050:85 , srv2016564585:85 , srv2019558295:85 , srv270732371:85 , srv308647732:85 , srv359763100:85 , srv73483525:85 , srv83634375:85 , srv901079285:85 , srv951887946:85 , srv987020808:85 , srv1281963546:86 , srv1975424187:86 , srv1992679668:86 , srv2068663833:86 , srv2126645116:86 , srv552703335:86 , srv592018905:86 , srv705621675:86 , srv816022746:86 , srv893116700:86 , srv967003987:86 , srv103551038:87 , srv1070222006:87 , srv1115817217:87 , srv1324973247:87 , srv1634611030:87 , srv2073936338:87 , srv348359454:87 , srv691416311:87 , srv949702897:87 , srv1036774794:88 , srv1248308380:88 , srv1301105846:88 , srv1567586419:88 , srv1629688621:88 , srv1650197709:88 , srv1885742251:88 , srv1973293353:88 , srv2070440284:88 , srv307464998:88 , srv374031600:88 , srv781096581:88 , srv878657741:88 , srv98555149:88 , srv1038904820:89 , srv1249811792:89 , srv1524440397:89 , srv426339324:89 , srv519024621:89 , srv695182215:89 , srv741623569:89 , srv835341360:89 , srv1248604078:90 , srv1743752527:90 , srv538003432:91 , srv559312461:91 , srv562786022:91 , srv869109690:91 , srv1013065194:92 , srv1968665493:92 , srv648923070:92 , srv692657735:93 , srv121627878:94 , srv497271460:462 } 2023-07-18 06:53:12,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 06:53:12,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1991520050=43, srv567003524=73, srv1278851582=16, srv521854817=67, srv1046767022=6, srv111166700=9, srv359763100=59, srv648923070=77, srv504866369=65, srv1301105846=18, srv1038904820=5, srv1248604078=14, srv1975424187=42, srv1669887750=32, srv2126645116=52, srv519024621=66, srv1248308380=13, srv1592168909=25, srv2070440284=49, srv1210972891=11, srv797091002=86, srv1650197709=31, srv605475450=76, srv1750483660=35, srv2016334220=45, srv307464998=56, srv1003778807=0, srv2068663833=48, srv73483525=83, srv967003987=97, srv103551038=2, srv1916443926=39, srv98555149=98, srv696188599=81, srv1324973247=20, srv781096581=85, srv1249811792=15, srv949702897=95, srv1973293353=41, srv691416311=78, srv426339324=63, srv878657741=91, srv1013065194=1, srv2016564585=46, srv1070222006=7, srv951887946=96, srv596422624=75, srv1629688621=29, srv1992679668=44, srv533344373=68, srv1634611030=30, srv1968665493=40, srv1036774794=4, srv1115817217=10, srv39932257=62, srv562786022=72, srv888461160=92, srv869109690=90, srv83634375=89, srv1322963617=19, srv1035986202=3, srv1605527308=27, srv2113258582=51, srv2019558295=47, srv2073936338=50, srv374031600=60, srv835341360=88, srv348359454=58, srv893116700=93, srv1769328462=36, srv552703335=70, srv1618978136=28, srv559312461=71, srv741623569=84, srv270732371=54, srv297322023=55, srv1694276587=33, srv695182215=80, srv538003432=69, srv592018905=74, srv1080157844=8, srv387237570=61, srv987020808=99, srv2144934416=53, srv692657735=79, srv901079285=94, srv1533979384=23, srv1567586419=24, srv1743752527=34, srv1814945854=37, srv1281963546=17, srv816022746=87, srv497271460=64, srv121627878=12, srv308647732=57, srv705621675=82, srv1524440397=22, srv1885742251=38, srv1445681738=21, srv1592921041=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 06:53:12,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 06:53:12,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 06:53:12,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 06:53:12,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 06:53:12,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 06:53:12,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 06:53:12,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 06:53:12,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 06:53:12,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 06:53:12,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 06:53:12,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 06:53:12,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 06:53:12,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 06:53:12,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 06:53:12,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 06:53:12,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 06:53:12,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 06:53:12,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 06:53:12,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 06:53:12,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 06:53:12,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 06:53:12,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 06:53:12,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 06:53:12,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 06:53:12,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 06:53:12,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 06:53:12,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 06:53:12,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 06:53:12,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 06:53:12,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 06:53:12,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 06:53:12,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 06:53:12,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 06:53:12,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 06:53:12,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 06:53:12,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 06:53:12,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 06:53:12,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 06:53:12,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 06:53:12,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 06:53:12,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 06:53:12,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 06:53:12,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 06:53:12,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 06:53:12,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 06:53:12,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 06:53:12,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 06:53:12,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 06:53:12,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 06:53:12,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 06:53:12,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 06:53:12,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 06:53:12,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 06:53:12,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 06:53:12,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 06:53:12,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 06:53:12,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 06:53:12,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 06:53:12,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 06:53:12,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 06:53:12,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 06:53:12,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 06:53:12,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 06:53:12,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 06:53:12,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 06:53:12,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 06:53:12,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 06:53:12,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 06:53:12,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 06:53:12,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 06:53:12,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 06:53:12,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 06:53:12,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 06:53:12,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 06:53:12,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 06:53:12,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 06:53:12,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 06:53:12,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 06:53:12,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 06:53:12,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 06:53:12,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 06:53:12,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 06:53:12,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 06:53:12,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 06:53:12,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 06:53:12,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 06:53:12,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 06:53:12,833 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04186421556896937, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05726462095020053, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04186421556896937, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17391666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.239, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 06:53:12,833 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-18 06:53:12,833 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1785490815602795, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04186421556896937, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05726462095020053, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04186421556896937, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17391666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.239, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-18 06:53:13,220 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 494 ms to try 5000 different iterations. Found a solution that moves 2629 regions; Going from a computed imbalance of 0.1785490815602795 to a new imbalance of 0.014418103707006129. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0244035981556616); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03450467201079284, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2921111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0244035981556616); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 06:53:13,231 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv692657735:78 , srv1013065194:79 , srv538003432:79 , srv1249811792:80 , srv1968665493:80 , srv121627878:81 , srv1743752527:81 , srv426339324:81 , srv559312461:81 , srv648923070:81 , srv835341360:81 , srv869109690:81 , srv1248604078:82 , srv1524440397:82 , srv374031600:82 , srv1324973247:83 , srv1973293353:83 , srv2070440284:83 , srv562786022:83 , srv878657741:83 , srv1038904820:84 , srv1070222006:84 , srv1629688621:84 , srv1634611030:84 , srv1650197709:84 , srv307464998:84 , srv949702897:84 , srv98555149:84 , srv103551038:85 , srv1036774794:85 , srv1248308380:85 , srv1281963546:85 , srv1301105846:85 , srv1567586419:85 , srv1885742251:85 , srv1975424187:85 , srv2019558295:85 , srv348359454:85 , srv359763100:85 , srv519024621:85 , srv691416311:85 , srv705621675:85 , srv741623569:85 , srv83634375:85 , srv901079285:85 , srv1115817217:86 , srv1210972891:86 , srv1322963617:86 , srv1618978136:86 , srv1750483660:86 , srv1992679668:86 , srv2016564585:86 , srv2068663833:86 , srv2126645116:86 , srv270732371:86 , srv39932257:86 , srv592018905:86 , srv781096581:86 , srv797091002:86 , srv816022746:86 , srv893116700:86 , srv951887946:86 , srv1003778807:87 , srv111166700:87 , srv1278851582:87 , srv1592921041:87 , srv1916443926:87 , srv1991520050:87 , srv2073936338:87 , srv2113258582:87 , srv297322023:87 , srv308647732:87 , srv521854817:87 , srv533344373:87 , srv552703335:87 , srv695182215:87 , srv73483525:87 , srv987020808:87 , srv1080157844:88 , srv1592168909:88 , srv1669887750:88 , srv2016334220:88 , srv2144934416:88 , srv504866369:88 , srv605475450:88 , srv696188599:88 , srv967003987:88 , srv1046767022:89 , srv1533979384:89 , srv1814945854:89 , srv596422624:89 , srv888461160:89 , srv1694276587:90 , srv1769328462:90 , srv387237570:90 , srv567003524:90 , srv1035986202:92 , srv1605527308:92 , srv1445681738:95 , srv497271460:529 } 2023-07-18 06:53:13,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 06:53:13,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1991520050=43, srv567003524=73, srv1278851582=16, srv521854817=67, srv1046767022=6, srv111166700=9, srv359763100=59, srv648923070=77, srv504866369=65, srv1301105846=18, srv1038904820=5, srv1248604078=14, srv1975424187=42, srv1669887750=32, srv2126645116=52, srv519024621=66, srv1248308380=13, srv1592168909=25, srv2070440284=49, srv1210972891=11, srv797091002=86, srv1650197709=31, srv605475450=76, srv1750483660=35, srv2016334220=45, srv307464998=56, srv1003778807=0, srv2068663833=48, srv73483525=83, srv967003987=97, srv103551038=2, srv1916443926=39, srv98555149=98, srv696188599=81, srv1324973247=20, srv781096581=85, srv1249811792=15, srv949702897=95, srv1973293353=41, srv691416311=78, srv426339324=63, srv878657741=91, srv1013065194=1, srv2016564585=46, srv1070222006=7, srv951887946=96, srv596422624=75, srv1629688621=29, srv1992679668=44, srv533344373=68, srv1634611030=30, srv1968665493=40, srv1036774794=4, srv1115817217=10, srv39932257=62, srv562786022=72, srv888461160=92, srv869109690=90, srv83634375=89, srv1322963617=19, srv1035986202=3, srv1605527308=27, srv2113258582=51, srv2019558295=47, srv2073936338=50, srv374031600=60, srv835341360=88, srv348359454=58, srv893116700=93, srv1769328462=36, srv552703335=70, srv1618978136=28, srv559312461=71, srv741623569=84, srv270732371=54, srv297322023=55, srv1694276587=33, srv695182215=80, srv538003432=69, srv592018905=74, srv1080157844=8, srv387237570=61, srv987020808=99, srv2144934416=53, srv692657735=79, srv901079285=94, srv1533979384=23, srv1567586419=24, srv1743752527=34, srv1814945854=37, srv1281963546=17, srv816022746=87, srv497271460=64, srv121627878=12, srv308647732=57, srv705621675=82, srv1524440397=22, srv1885742251=38, srv1445681738=21, srv1592921041=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 06:53:13,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 06:53:13,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 06:53:13,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 06:53:13,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 06:53:13,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 06:53:13,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 06:53:13,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 06:53:13,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 06:53:13,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 06:53:13,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 06:53:13,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 06:53:13,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 06:53:13,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 06:53:13,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 06:53:13,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 06:53:13,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 06:53:13,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 06:53:13,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 06:53:13,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 06:53:13,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 06:53:13,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 06:53:13,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 06:53:13,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 06:53:13,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 06:53:13,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 06:53:13,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 06:53:13,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 06:53:13,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 06:53:13,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 06:53:13,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 06:53:13,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 06:53:13,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 06:53:13,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 06:53:13,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 06:53:13,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 06:53:13,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 06:53:13,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 06:53:13,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 06:53:13,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 06:53:13,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 06:53:13,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 06:53:13,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 06:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 06:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 06:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 06:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 06:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 06:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 06:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 06:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 06:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 06:53:13,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 06:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 06:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 06:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 06:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 06:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 06:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 06:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 06:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 06:53:13,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 06:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 06:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 06:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 06:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 06:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 06:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 06:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 06:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 06:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 06:53:13,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 06:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 06:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 06:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 06:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 06:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 06:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 06:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 06:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 06:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 06:53:13,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 06:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 06:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 06:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 06:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 06:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 06:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 06:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 06:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 06:53:13,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 06:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 06:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 06:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 06:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 06:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 06:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 06:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 06:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 06:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 06:53:13,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 06:53:13,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 06:53:13,399 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0244035981556616); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03450467201079284, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0244035981556616); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 06:53:13,400 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-18 06:53:13,400 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014399695055550134, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0244035981556616); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03450467201079284, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0244035981556616); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-18 06:53:13,740 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 461 ms to try 5000 different iterations. Found a solution that moves 956 regions; Going from a computed imbalance of 0.014399695055550134 to a new imbalance of 5.907201139298467E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006993485647474662); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01059402876939547); MoveCostFunction : (multiplier=7.0, imbalance=0.10622222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006993485647474662); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 06:53:13,742 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv695182215:81 , srv121627878:82 , srv1445681738:82 , srv1533979384:82 , srv1605527308:83 , srv1694276587:83 , srv2073936338:83 , srv741623569:83 , srv967003987:83 , srv1036774794:84 , srv1038904820:84 , srv1115817217:84 , srv1248308380:84 , srv1301105846:84 , srv387237570:84 , srv519024621:84 , srv552703335:84 , srv648923070:84 , srv1003778807:85 , srv1013065194:85 , srv1035986202:85 , srv1248604078:85 , srv1567586419:85 , srv1592168909:85 , srv1650197709:85 , srv1669887750:85 , srv1769328462:85 , srv1814945854:85 , srv1885742251:85 , srv1991520050:85 , srv1992679668:85 , srv2016334220:85 , srv2070440284:85 , srv2126645116:85 , srv2144934416:85 , srv308647732:85 , srv504866369:85 , srv559312461:85 , srv562786022:85 , srv592018905:85 , srv692657735:85 , srv696188599:85 , srv73483525:85 , srv781096581:85 , srv816022746:85 , srv869109690:85 , srv103551038:86 , srv1046767022:86 , srv1070222006:86 , srv1080157844:86 , srv111166700:86 , srv1524440397:86 , srv1592921041:86 , srv1618978136:86 , srv1629688621:86 , srv1968665493:86 , srv1973293353:86 , srv1975424187:86 , srv2068663833:86 , srv2113258582:86 , srv270732371:86 , srv297322023:86 , srv307464998:86 , srv348359454:86 , srv567003524:86 , srv596422624:86 , srv691416311:86 , srv705621675:86 , srv878657741:86 , srv893116700:86 , srv951887946:86 , srv98555149:86 , srv987020808:86 , srv1210972891:87 , srv1278851582:87 , srv1281963546:87 , srv1634611030:87 , srv1743752527:87 , srv1750483660:87 , srv1916443926:87 , srv2016564585:87 , srv2019558295:87 , srv359763100:87 , srv39932257:87 , srv426339324:87 , srv533344373:87 , srv538003432:87 , srv605475450:87 , srv83634375:87 , srv888461160:87 , srv949702897:87 , srv1249811792:88 , srv1322963617:88 , srv1324973247:88 , srv374031600:88 , srv521854817:88 , srv835341360:88 , srv901079285:88 , srv797091002:89 , srv497271460:529 } 2023-07-18 06:53:13,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 06:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1991520050=43, srv567003524=73, srv1278851582=16, srv521854817=67, srv1046767022=6, srv111166700=9, srv359763100=59, srv648923070=77, srv504866369=65, srv1301105846=18, srv1038904820=5, srv1248604078=14, srv1975424187=42, srv1669887750=32, srv2126645116=52, srv519024621=66, srv1248308380=13, srv1592168909=25, srv2070440284=49, srv1210972891=11, srv797091002=86, srv1650197709=31, srv605475450=76, srv1750483660=35, srv2016334220=45, srv307464998=56, srv1003778807=0, srv2068663833=48, srv73483525=83, srv967003987=97, srv103551038=2, srv1916443926=39, srv98555149=98, srv696188599=81, srv1324973247=20, srv781096581=85, srv1249811792=15, srv949702897=95, srv1973293353=41, srv691416311=78, srv426339324=63, srv878657741=91, srv1013065194=1, srv2016564585=46, srv1070222006=7, srv951887946=96, srv596422624=75, srv1629688621=29, srv1992679668=44, srv533344373=68, srv1634611030=30, srv1968665493=40, srv1036774794=4, srv1115817217=10, srv39932257=62, srv562786022=72, srv888461160=92, srv869109690=90, srv83634375=89, srv1322963617=19, srv1035986202=3, srv1605527308=27, srv2113258582=51, srv2019558295=47, srv2073936338=50, srv374031600=60, srv835341360=88, srv348359454=58, srv893116700=93, srv1769328462=36, srv552703335=70, srv1618978136=28, srv559312461=71, srv741623569=84, srv270732371=54, srv297322023=55, srv1694276587=33, srv695182215=80, srv538003432=69, srv592018905=74, srv1080157844=8, srv387237570=61, srv987020808=99, srv2144934416=53, srv692657735=79, srv901079285=94, srv1533979384=23, srv1567586419=24, srv1743752527=34, srv1814945854=37, srv1281963546=17, srv816022746=87, srv497271460=64, srv121627878=12, srv308647732=57, srv705621675=82, srv1524440397=22, srv1885742251=38, srv1445681738=21, srv1592921041=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 06:53:13,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 06:53:13,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 06:53:13,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 06:53:13,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 06:53:13,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 06:53:13,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 06:53:13,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 06:53:13,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 06:53:13,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 06:53:13,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 06:53:13,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 06:53:13,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 06:53:13,886 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006993485647474662); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01059402876939547); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006993485647474662); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 06:53:13,886 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-18 06:53:13,886 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.840260588549386E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006993485647474662); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01059402876939547); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006993485647474662); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-18 06:53:14,269 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 483 ms to try 5000 different iterations. Found a solution that moves 267 regions; Going from a computed imbalance of 5.840260588549386E-4 to a new imbalance of 1.9343247189750356E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.029666666666666668, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 06:53:14,270 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2068663833:83 , srv562786022:83 , srv781096581:83 , srv797091002:83 , srv1003778807:84 , srv103551038:84 , srv1035986202:84 , srv1036774794:84 , srv1046767022:84 , srv1210972891:84 , srv1278851582:84 , srv1281963546:84 , srv1322963617:84 , srv1324973247:84 , srv1445681738:84 , srv1567586419:84 , srv1592168909:84 , srv1605527308:84 , srv1629688621:84 , srv1634611030:84 , srv1743752527:84 , srv1750483660:84 , srv1814945854:84 , srv1885742251:84 , srv1916443926:84 , srv1968665493:84 , srv1991520050:84 , srv2016564585:84 , srv2126645116:84 , srv307464998:84 , srv348359454:84 , srv374031600:84 , srv387237570:84 , srv519024621:84 , srv533344373:84 , srv596422624:84 , srv605475450:84 , srv691416311:84 , srv695182215:84 , srv835341360:84 , srv893116700:84 , srv901079285:84 , srv949702897:84 , srv98555149:84 , srv987020808:84 , srv1038904820:85 , srv1070222006:85 , srv1080157844:85 , srv111166700:85 , srv1115817217:85 , srv121627878:85 , srv1248604078:85 , srv1301105846:85 , srv1524440397:85 , srv1592921041:85 , srv1618978136:85 , srv1650197709:85 , srv1669887750:85 , srv1769328462:85 , srv1975424187:85 , srv1992679668:85 , srv2016334220:85 , srv2019558295:85 , srv2073936338:85 , srv2113258582:85 , srv2144934416:85 , srv270732371:85 , srv297322023:85 , srv308647732:85 , srv359763100:85 , srv39932257:85 , srv504866369:85 , srv521854817:85 , srv538003432:85 , srv552703335:85 , srv559312461:85 , srv567003524:85 , srv592018905:85 , srv648923070:85 , srv696188599:85 , srv705621675:85 , srv73483525:85 , srv741623569:85 , srv816022746:85 , srv83634375:85 , srv878657741:85 , srv888461160:85 , srv951887946:85 , srv967003987:85 , srv1248308380:86 , srv1249811792:86 , srv1533979384:86 , srv1694276587:86 , srv1973293353:86 , srv2070440284:86 , srv426339324:86 , srv692657735:86 , srv869109690:86 , srv1013065194:87 , srv497271460:623 } 2023-07-18 06:53:14,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 06:53:14,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1991520050=43, srv567003524=73, srv1278851582=16, srv521854817=67, srv1046767022=6, srv111166700=9, srv359763100=59, srv648923070=77, srv504866369=65, srv1301105846=18, srv1038904820=5, srv1248604078=14, srv1975424187=42, srv1669887750=32, srv2126645116=52, srv519024621=66, srv1248308380=13, srv1592168909=25, srv2070440284=49, srv1210972891=11, srv797091002=86, srv1650197709=31, srv605475450=76, srv1750483660=35, srv2016334220=45, srv307464998=56, srv1003778807=0, srv2068663833=48, srv73483525=83, srv967003987=97, srv103551038=2, srv1916443926=39, srv98555149=98, srv696188599=81, srv1324973247=20, srv781096581=85, srv1249811792=15, srv949702897=95, srv1973293353=41, srv691416311=78, srv426339324=63, srv878657741=91, srv1013065194=1, srv2016564585=46, srv1070222006=7, srv951887946=96, srv596422624=75, srv1629688621=29, srv1992679668=44, srv533344373=68, srv1634611030=30, srv1968665493=40, srv1036774794=4, srv1115817217=10, srv39932257=62, srv562786022=72, srv888461160=92, srv869109690=90, srv83634375=89, srv1322963617=19, srv1035986202=3, srv1605527308=27, srv2113258582=51, srv2019558295=47, srv2073936338=50, srv374031600=60, srv835341360=88, srv348359454=58, srv893116700=93, srv1769328462=36, srv552703335=70, srv1618978136=28, srv559312461=71, srv741623569=84, srv270732371=54, srv297322023=55, srv1694276587=33, srv695182215=80, srv538003432=69, srv592018905=74, srv1080157844=8, srv387237570=61, srv987020808=99, srv2144934416=53, srv692657735=79, srv901079285=94, srv1533979384=23, srv1567586419=24, srv1743752527=34, srv1814945854=37, srv1281963546=17, srv816022746=87, srv497271460=64, srv121627878=12, srv308647732=57, srv705621675=82, srv1524440397=22, srv1885742251=38, srv1445681738=21, srv1592921041=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 06:53:14,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 06:53:14,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 06:53:14,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 06:53:14,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 06:53:14,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 06:53:14,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 06:53:14,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 06:53:14,403 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.747367323054488E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 06:53:14,403 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv2068663833:83 , srv562786022:83 , srv781096581:83 , srv797091002:83 , srv1003778807:84 , srv103551038:84 , srv1035986202:84 , srv1036774794:84 , srv1046767022:84 , srv1210972891:84 , srv1278851582:84 , srv1281963546:84 , srv1322963617:84 , srv1324973247:84 , srv1445681738:84 , srv1567586419:84 , srv1592168909:84 , srv1605527308:84 , srv1629688621:84 , srv1634611030:84 , srv1743752527:84 , srv1750483660:84 , srv1814945854:84 , srv1885742251:84 , srv1916443926:84 , srv1968665493:84 , srv1991520050:84 , srv2016564585:84 , srv2126645116:84 , srv307464998:84 , srv348359454:84 , srv374031600:84 , srv387237570:84 , srv519024621:84 , srv533344373:84 , srv596422624:84 , srv605475450:84 , srv691416311:84 , srv695182215:84 , srv835341360:84 , srv893116700:84 , srv901079285:84 , srv949702897:84 , srv98555149:84 , srv987020808:84 , srv1038904820:85 , srv1070222006:85 , srv1080157844:85 , srv111166700:85 , srv1115817217:85 , srv121627878:85 , srv1248604078:85 , srv1301105846:85 , srv1524440397:85 , srv1592921041:85 , srv1618978136:85 , srv1650197709:85 , srv1669887750:85 , srv1769328462:85 , srv1975424187:85 , srv1992679668:85 , srv2016334220:85 , srv2019558295:85 , srv2073936338:85 , srv2113258582:85 , srv2144934416:85 , srv270732371:85 , srv297322023:85 , srv308647732:85 , srv359763100:85 , srv39932257:85 , srv504866369:85 , srv521854817:85 , srv538003432:85 , srv552703335:85 , srv559312461:85 , srv567003524:85 , srv592018905:85 , srv648923070:85 , srv696188599:85 , srv705621675:85 , srv73483525:85 , srv741623569:85 , srv816022746:85 , srv83634375:85 , srv878657741:85 , srv888461160:85 , srv951887946:85 , srv967003987:85 , srv1248308380:86 , srv1249811792:86 , srv1533979384:86 , srv1694276587:86 , srv1973293353:86 , srv2070440284:86 , srv426339324:86 , srv692657735:86 , srv869109690:86 , srv1013065194:87 , srv497271460:623 } 2023-07-18 06:53:14,433 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=308 (was 308), ProcessCount=172 (was 172), AvailableMemoryMB=6740 (was 7528) 2023-07-18 06:53:14,442 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=308, ProcessCount=172, AvailableMemoryMB=6740 2023-07-18 06:53:14,443 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-18 06:53:14,443 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-07-18 06:53:14,456 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1138268114:84 , srv1164196991:84 , srv1256930443:84 , srv1297074157:84 , srv1308317263:84 , srv1447422267:84 , srv145957549:84 , srv1480495995:84 , srv1535673321:84 , srv1562230933:84 , srv1654669397:84 , srv1806177481:84 , srv1825117532:84 , srv1847648504:84 , srv1891566219:84 , srv1956487111:84 , srv199069963:84 , srv2005589315:84 , srv2135106166:84 , srv285195005:84 , srv355255090:84 , srv398560578:84 , srv408638364:84 , srv491275972:84 , srv64290582:84 , srv75274485:84 , srv802066832:84 , srv82209123:84 , srv888365073:84 , srv1258041574:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-07-18 06:53:14,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 06:53:14,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv285195005=20, srv1447422267=6, srv2135106166=19, srv1654669397=11, srv1825117532=13, srv2005589315=18, srv1562230933=10, srv355255090=21, srv82209123=28, srv1806177481=12, srv145957549=7, srv408638364=23, srv888365073=29, srv1847648504=14, srv1891566219=15, srv398560578=22, srv1258041574=3, srv199069963=17, srv802066832=27, srv1480495995=8, srv1297074157=4, srv1308317263=5, srv75274485=26, srv1256930443=2, srv64290582=25, srv1138268114=0, srv491275972=24, srv1956487111=16, srv1535673321=9, srv1164196991=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 06:53:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 06:53:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 06:53:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 06:53:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 06:53:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 06:53:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 06:53:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 06:53:14,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 06:53:14,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 06:53:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 06:53:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 06:53:14,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-18 06:53:14,501 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0773647015121421, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 06:53:14,502 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912436816912555, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0773647015121421, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-07-18 06:55:14,489 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5461707 different iterations. Found a solution that moves 1874 regions; Going from a computed imbalance of 0.9912436816912555 to a new imbalance of 4.374009487579354E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6940740740740741, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 06:55:14,496 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1138268114:90 , srv1164196991:90 , srv1256930443:90 , srv1258041574:90 , srv1297074157:90 , srv1308317263:90 , srv1447422267:90 , srv145957549:90 , srv1480495995:90 , srv1535673321:90 , srv1562230933:90 , srv1654669397:90 , srv1806177481:90 , srv1825117532:90 , srv1847648504:90 , srv1891566219:90 , srv1956487111:90 , srv199069963:90 , srv2005589315:90 , srv2135106166:90 , srv285195005:90 , srv355255090:90 , srv398560578:90 , srv408638364:90 , srv491275972:90 , srv64290582:90 , srv75274485:90 , srv802066832:90 , srv82209123:90 , srv888365073:90 } 2023-07-18 06:55:14,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 06:55:14,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv285195005=20, srv1447422267=6, srv2135106166=19, srv1654669397=11, srv1825117532=13, srv2005589315=18, srv1562230933=10, srv355255090=21, srv82209123=28, srv1806177481=12, srv145957549=7, srv408638364=23, srv888365073=29, srv1847648504=14, srv1891566219=15, srv398560578=22, srv1258041574=3, srv199069963=17, srv802066832=27, srv1480495995=8, srv1297074157=4, srv1308317263=5, srv75274485=26, srv1256930443=2, srv64290582=25, srv1138268114=0, srv491275972=24, srv1956487111=16, srv1535673321=9, srv1164196991=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 06:55:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 06:55:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 06:55:14,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 06:55:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 06:55:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 06:55:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 06:55:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 06:55:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 06:55:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 06:55:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 06:55:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 06:55:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 06:55:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 06:55:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 06:55:14,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 06:55:14,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-18 06:55:14,518 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 06:55:14,518 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1138268114:90 , srv1164196991:90 , srv1256930443:90 , srv1258041574:90 , srv1297074157:90 , srv1308317263:90 , srv1447422267:90 , srv145957549:90 , srv1480495995:90 , srv1535673321:90 , srv1562230933:90 , srv1654669397:90 , srv1806177481:90 , srv1825117532:90 , srv1847648504:90 , srv1891566219:90 , srv1956487111:90 , srv199069963:90 , srv2005589315:90 , srv2135106166:90 , srv285195005:90 , srv355255090:90 , srv398560578:90 , srv408638364:90 , srv491275972:90 , srv64290582:90 , srv75274485:90 , srv802066832:90 , srv82209123:90 , srv888365073:90 } 2023-07-18 06:55:14,531 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=202 (was 308), ProcessCount=172 (was 172), AvailableMemoryMB=5920 (was 6740)