2023-10-16 11:53:27,104 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-16 11:53:27,393 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-16 11:53:27,504 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-16 11:53:27,571 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-16 11:53:27,579 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-16 11:53:27,592 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=336, ProcessCount=170, AvailableMemoryMB=6029 2023-10-16 11:53:27,594 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-16 11:53:27,595 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-16 11:53:27,705 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1017850351:84 , srv1025931700:84 , srv1053559404:84 , srv105999388:84 , srv1100422842:84 , srv1131201566:84 , srv1150761673:84 , srv1175347504:84 , srv1221427148:84 , srv1223435392:84 , srv1235216844:84 , srv1242736848:84 , srv1250365830:84 , srv1263466912:84 , srv1298469133:84 , srv1306271367:84 , srv1317519828:84 , srv136877463:84 , srv137534181:84 , srv1387536904:84 , srv1406260689:84 , srv1434007054:84 , srv1506179239:84 , srv153469746:84 , srv155106741:84 , srv1631705580:84 , srv1647452802:84 , srv1656220437:84 , srv1672118627:84 , srv1685820486:84 , srv1692867915:84 , srv1710015818:84 , srv1727977942:84 , srv1744650430:84 , srv1755880675:84 , srv1768375928:84 , srv1832986359:84 , srv1908068330:84 , srv1917604044:84 , srv1956485304:84 , srv1971452360:84 , srv1973549731:84 , srv2006150489:84 , srv2054562125:84 , srv2065572237:84 , srv2071710822:84 , srv2073345365:84 , srv2095899647:84 , srv2106961072:84 , srv2120402559:84 , srv2120511941:84 , srv230623414:84 , srv235560416:84 , srv268833657:84 , srv269177671:84 , srv279686493:84 , srv31762935:84 , srv349817455:84 , srv436279762:84 , srv477113741:84 , srv498600364:84 , srv517799152:84 , srv535861304:84 , srv552621924:84 , srv553298073:84 , srv560625111:84 , srv571387681:84 , srv579845024:84 , srv583181237:84 , srv584360703:84 , srv588763407:84 , srv614892327:84 , srv65116946:84 , srv686307461:84 , srv720417579:84 , srv723107193:84 , srv738511247:84 , srv762189627:84 , srv767909299:84 , srv770888256:84 , srv773187910:84 , srv784923252:84 , srv798539124:84 , srv831554074:84 , srv83905965:84 , srv844744509:84 , srv851622295:84 , srv866375873:84 , srv894732152:84 , srv914613475:84 , srv914898664:84 , srv922005314:84 , srv932424714:84 , srv939352392:84 , srv957800658:84 , srv96723364:84 , srv969218489:84 , srv969739394:84 , srv977132599:84 , srv340660885:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-16 11:53:27,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 11:53:27,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1263466912=13, srv1973549731=41, srv517799152=62, srv1025931700=1, srv137534181=18, srv436279762=59, srv773187910=81, srv1727977942=32, srv552621924=64, srv1956485304=39, srv831554074=84, srv2106961072=48, srv571387681=67, srv770888256=80, srv1406260689=20, srv1298469133=14, srv268833657=53, srv1908068330=37, srv1017850351=0, srv153469746=23, srv279686493=55, srv1053559404=2, srv1223435392=9, srv2120511941=50, srv1755880675=34, srv1306271367=15, srv579845024=68, srv1631705580=25, srv686307461=74, srv2071710822=45, srv583181237=69, srv230623414=51, srv477113741=60, srv2095899647=47, srv614892327=72, srv939352392=94, srv553298073=65, srv340660885=57, srv738511247=77, srv1506179239=22, srv1250365830=12, srv1175347504=7, srv155106741=24, srv535861304=63, srv498600364=61, srv969739394=98, srv1672118627=28, srv2065572237=44, srv957800658=95, srv844744509=86, srv584360703=70, srv932424714=93, srv560625111=66, srv588763407=71, srv1832986359=36, srv65116946=73, srv1100422842=4, srv1387536904=19, srv1917604044=38, srv767909299=79, srv1317519828=16, srv136877463=17, srv235560416=52, srv784923252=82, srv914898664=91, srv1692867915=30, srv2120402559=49, srv1685820486=29, srv1971452360=40, srv866375873=88, srv1434007054=21, srv96723364=96, srv1744650430=33, srv83905965=85, srv914613475=90, srv1647452802=26, srv1242736848=11, srv1221427148=8, srv1768375928=35, srv2054562125=43, srv2073345365=46, srv1235216844=10, srv31762935=56, srv977132599=99, srv2006150489=42, srv762189627=78, srv1131201566=5, srv723107193=76, srv1710015818=31, srv1656220437=27, srv269177671=54, srv105999388=3, srv349817455=58, srv894732152=89, srv1150761673=6, srv922005314=92, srv851622295=87, srv720417579=75, srv798539124=83, srv969218489=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 11:53:27,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 11:53:27,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 11:53:27,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 11:53:27,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 11:53:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 11:53:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 11:53:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 11:53:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 11:53:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 11:53:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 11:53:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 11:53:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 11:53:27,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 11:53:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 11:53:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 11:53:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 11:53:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 11:53:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 11:53:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 11:53:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 11:53:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 11:53:27,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 11:53:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 11:53:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 11:53:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 11:53:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 11:53:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 11:53:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 11:53:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 11:53:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 11:53:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-16 11:53:27,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-16 11:53:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-16 11:53:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-16 11:53:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-16 11:53:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-16 11:53:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-16 11:53:27,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-16 11:53:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-16 11:53:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-16 11:53:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-16 11:53:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-16 11:53:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-16 11:53:27,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-16 11:53:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-16 11:53:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-16 11:53:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-16 11:53:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-16 11:53:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-16 11:53:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-16 11:53:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-16 11:53:27,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-16 11:53:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-16 11:53:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-16 11:53:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-16 11:53:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-16 11:53:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-16 11:53:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-16 11:53:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-16 11:53:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-16 11:53:27,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-16 11:53:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-16 11:53:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-16 11:53:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-16 11:53:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-16 11:53:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-16 11:53:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-16 11:53:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-16 11:53:27,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-16 11:53:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-16 11:53:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-16 11:53:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-16 11:53:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-16 11:53:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-16 11:53:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-16 11:53:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-16 11:53:27,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-16 11:53:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-16 11:53:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-16 11:53:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-16 11:53:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-16 11:53:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-16 11:53:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-16 11:53:27,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-16 11:53:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-16 11:53:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-16 11:53:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-16 11:53:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-16 11:53:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-16 11:53:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-16 11:53:27,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-16 11:53:27,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-16 11:53:27,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-16 11:53:27,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-16 11:53:27,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-16 11:53:27,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-16 11:53:27,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-16 11:53:27,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-16 11:53:27,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-16 11:53:27,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-16 11:53:27,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-16 11:53:28,054 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:53:28,054 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-16 11:53:28,055 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-16 11:53:28,550 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 700 ms to try 5000 different iterations. Found a solution that moves 4698 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17782183388506317. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03826483412827236, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057246979051736376, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.522, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03826483412827236, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:53:28,574 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1242736848:81 , srv105999388:83 , srv1768375928:83 , srv2120402559:83 , srv614892327:83 , srv798539124:83 , srv1306271367:84 , srv1956485304:84 , srv2106961072:84 , srv235560416:84 , srv279686493:84 , srv436279762:84 , srv583181237:84 , srv939352392:84 , srv977132599:84 , srv1131201566:85 , srv1175347504:85 , srv1221427148:85 , srv1317519828:85 , srv137534181:85 , srv153469746:85 , srv1647452802:85 , srv1656220437:85 , srv1692867915:85 , srv1971452360:85 , srv2006150489:85 , srv2065572237:85 , srv2073345365:85 , srv2120511941:85 , srv552621924:85 , srv571387681:85 , srv579845024:85 , srv584360703:85 , srv686307461:85 , srv770888256:85 , srv851622295:85 , srv894732152:85 , srv969739394:85 , srv1017850351:86 , srv1223435392:86 , srv1387536904:86 , srv1406260689:86 , srv1506179239:86 , srv1672118627:86 , srv1710015818:86 , srv1917604044:86 , srv2095899647:86 , srv517799152:86 , srv535861304:86 , srv553298073:86 , srv762189627:86 , srv767909299:86 , srv784923252:86 , srv831554074:86 , srv922005314:86 , srv1263466912:87 , srv1298469133:87 , srv136877463:87 , srv1631705580:87 , srv1727977942:87 , srv1755880675:87 , srv1973549731:87 , srv2054562125:87 , srv269177671:87 , srv588763407:87 , srv83905965:87 , srv914613475:87 , srv957800658:87 , srv96723364:87 , srv969218489:87 , srv1100422842:88 , srv1235216844:88 , srv155106741:88 , srv1832986359:88 , srv2071710822:88 , srv230623414:88 , srv31762935:88 , srv349817455:88 , srv477113741:88 , srv560625111:88 , srv65116946:88 , srv720417579:88 , srv723107193:88 , srv844744509:88 , srv866375873:88 , srv1053559404:89 , srv1150761673:89 , srv268833657:89 , srv498600364:89 , srv932424714:89 , srv1908068330:90 , srv914898664:90 , srv1025931700:91 , srv1250365830:91 , srv1744650430:91 , srv1685820486:92 , srv773187910:92 , srv738511247:95 , srv1434007054:99 , srv340660885:430 } 2023-10-16 11:53:28,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 11:53:28,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1263466912=13, srv1973549731=41, srv517799152=62, srv1025931700=1, srv137534181=18, srv436279762=59, srv773187910=81, srv1727977942=32, srv552621924=64, srv1956485304=39, srv831554074=84, srv2106961072=48, srv571387681=67, srv770888256=80, srv1406260689=20, srv1298469133=14, srv268833657=53, srv1908068330=37, srv1017850351=0, srv153469746=23, srv279686493=55, srv1053559404=2, srv1223435392=9, srv2120511941=50, srv1755880675=34, srv1306271367=15, srv579845024=68, srv1631705580=25, srv686307461=74, srv2071710822=45, srv583181237=69, srv230623414=51, srv477113741=60, srv2095899647=47, srv614892327=72, srv939352392=94, srv553298073=65, srv340660885=57, srv738511247=77, srv1506179239=22, srv1250365830=12, srv1175347504=7, srv155106741=24, srv535861304=63, srv498600364=61, srv969739394=98, srv1672118627=28, srv2065572237=44, srv957800658=95, srv844744509=86, srv584360703=70, srv932424714=93, srv560625111=66, srv588763407=71, srv1832986359=36, srv65116946=73, srv1100422842=4, srv1387536904=19, srv1917604044=38, srv767909299=79, srv1317519828=16, srv136877463=17, srv235560416=52, srv784923252=82, srv914898664=91, srv1692867915=30, srv2120402559=49, srv1685820486=29, srv1971452360=40, srv866375873=88, srv1434007054=21, srv96723364=96, srv1744650430=33, srv83905965=85, srv914613475=90, srv1647452802=26, srv1242736848=11, srv1221427148=8, srv1768375928=35, srv2054562125=43, srv2073345365=46, srv1235216844=10, srv31762935=56, srv977132599=99, srv2006150489=42, srv762189627=78, srv1131201566=5, srv723107193=76, srv1710015818=31, srv1656220437=27, srv269177671=54, srv105999388=3, srv349817455=58, srv894732152=89, srv1150761673=6, srv922005314=92, srv851622295=87, srv720417579=75, srv798539124=83, srv969218489=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 11:53:28,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 11:53:28,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 11:53:28,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 11:53:28,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 11:53:28,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 11:53:28,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 11:53:28,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 11:53:28,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 11:53:28,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 11:53:28,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 11:53:28,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 11:53:28,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 11:53:28,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 11:53:28,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 11:53:28,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 11:53:28,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 11:53:28,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 11:53:28,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 11:53:28,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 11:53:28,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 11:53:28,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 11:53:28,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 11:53:28,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 11:53:28,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 11:53:28,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 11:53:28,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 11:53:28,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 11:53:28,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 11:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 11:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 11:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-16 11:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-16 11:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-16 11:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-16 11:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-16 11:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-16 11:53:28,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-16 11:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-16 11:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-16 11:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-16 11:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-16 11:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-16 11:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-16 11:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-16 11:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-16 11:53:28,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-16 11:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-16 11:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-16 11:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-16 11:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-16 11:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-16 11:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-16 11:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-16 11:53:28,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-16 11:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-16 11:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-16 11:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-16 11:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-16 11:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-16 11:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-16 11:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-16 11:53:28,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-16 11:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-16 11:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-16 11:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-16 11:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-16 11:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-16 11:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-16 11:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-16 11:53:28,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-16 11:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-16 11:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-16 11:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-16 11:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-16 11:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-16 11:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-16 11:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-16 11:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-16 11:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-16 11:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-16 11:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-16 11:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-16 11:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-16 11:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-16 11:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-16 11:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-16 11:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-16 11:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-16 11:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-16 11:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-16 11:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-16 11:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-16 11:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-16 11:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-16 11:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-16 11:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-16 11:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-16 11:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-16 11:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-16 11:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-16 11:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-16 11:53:28,818 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03826483412827236, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057246979051736376, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03826483412827236, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:53:28,818 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-16 11:53:28,819 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17778893778596072, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03826483412827236, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057246979051736376, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03826483412827236, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-16 11:53:29,117 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 482 ms to try 5000 different iterations. Found a solution that moves 2606 regions; Going from a computed imbalance of 0.17778893778596072 to a new imbalance of 0.016489407911103315. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019145611330566067); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03364834318176254, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28955555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019145611330566067); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:53:29,126 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1434007054:76 , srv738511247:77 , srv1685820486:79 , srv773187910:79 , srv1250365830:80 , srv1744650430:80 , srv1908068330:80 , srv1150761673:82 , srv1025931700:83 , srv1053559404:83 , srv1727977942:83 , srv1832986359:83 , srv230623414:83 , srv477113741:83 , srv914898664:83 , srv932424714:83 , srv1100422842:84 , srv1235216844:84 , srv1298469133:84 , srv136877463:84 , srv1406260689:84 , srv155106741:84 , srv1631705580:84 , srv1973549731:84 , srv268833657:84 , srv723107193:84 , srv914613475:84 , srv1223435392:85 , srv1692867915:85 , srv1755880675:85 , srv2071710822:85 , srv2095899647:85 , srv2120511941:85 , srv269177671:85 , srv349817455:85 , srv498600364:85 , srv553298073:85 , srv65116946:85 , srv784923252:85 , srv866375873:85 , srv957800658:85 , srv969218489:85 , srv1175347504:86 , srv1221427148:86 , srv1263466912:86 , srv1387536904:86 , srv153469746:86 , srv1672118627:86 , srv1710015818:86 , srv1917604044:86 , srv1971452360:86 , srv2006150489:86 , srv31762935:86 , srv552621924:86 , srv579845024:86 , srv770888256:86 , srv831554074:86 , srv83905965:86 , srv851622295:86 , srv96723364:86 , srv105999388:87 , srv2073345365:87 , srv2106961072:87 , srv560625111:87 , srv583181237:87 , srv588763407:87 , srv686307461:87 , srv720417579:87 , srv762189627:87 , srv844744509:87 , srv894732152:87 , srv939352392:87 , srv969739394:87 , srv1017850351:88 , srv1131201566:88 , srv137534181:88 , srv2054562125:88 , srv2065572237:88 , srv2120402559:88 , srv279686493:88 , srv436279762:88 , srv535861304:88 , srv584360703:88 , srv767909299:88 , srv922005314:88 , srv1306271367:89 , srv1506179239:89 , srv1647452802:89 , srv1656220437:89 , srv1768375928:89 , srv235560416:89 , srv517799152:89 , srv614892327:89 , srv798539124:89 , srv1317519828:90 , srv1956485304:90 , srv977132599:90 , srv571387681:91 , srv1242736848:93 , srv340660885:514 } 2023-10-16 11:53:29,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 11:53:29,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1263466912=13, srv1973549731=41, srv517799152=62, srv1025931700=1, srv137534181=18, srv436279762=59, srv773187910=81, srv1727977942=32, srv552621924=64, srv1956485304=39, srv831554074=84, srv2106961072=48, srv571387681=67, srv770888256=80, srv1406260689=20, srv1298469133=14, srv268833657=53, srv1908068330=37, srv1017850351=0, srv153469746=23, srv279686493=55, srv1053559404=2, srv1223435392=9, srv2120511941=50, srv1755880675=34, srv1306271367=15, srv579845024=68, srv1631705580=25, srv686307461=74, srv2071710822=45, srv583181237=69, srv230623414=51, srv477113741=60, srv2095899647=47, srv614892327=72, srv939352392=94, srv553298073=65, srv340660885=57, srv738511247=77, srv1506179239=22, srv1250365830=12, srv1175347504=7, srv155106741=24, srv535861304=63, srv498600364=61, srv969739394=98, srv1672118627=28, srv2065572237=44, srv957800658=95, srv844744509=86, srv584360703=70, srv932424714=93, srv560625111=66, srv588763407=71, srv1832986359=36, srv65116946=73, srv1100422842=4, srv1387536904=19, srv1917604044=38, srv767909299=79, srv1317519828=16, srv136877463=17, srv235560416=52, srv784923252=82, srv914898664=91, srv1692867915=30, srv2120402559=49, srv1685820486=29, srv1971452360=40, srv866375873=88, srv1434007054=21, srv96723364=96, srv1744650430=33, srv83905965=85, srv914613475=90, srv1647452802=26, srv1242736848=11, srv1221427148=8, srv1768375928=35, srv2054562125=43, srv2073345365=46, srv1235216844=10, srv31762935=56, srv977132599=99, srv2006150489=42, srv762189627=78, srv1131201566=5, srv723107193=76, srv1710015818=31, srv1656220437=27, srv269177671=54, srv105999388=3, srv349817455=58, srv894732152=89, srv1150761673=6, srv922005314=92, srv851622295=87, srv720417579=75, srv798539124=83, srv969218489=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 11:53:29,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 11:53:29,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-16 11:53:29,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-16 11:53:29,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-16 11:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-16 11:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-16 11:53:29,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-16 11:53:29,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-16 11:53:29,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-16 11:53:29,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-16 11:53:29,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-16 11:53:29,404 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019145611330566067); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03364834318176254, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019145611330566067); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:53:29,404 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-16 11:53:29,404 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016471160309089496, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019145611330566067); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03364834318176254, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019145611330566067); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-16 11:53:29,670 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 484 ms to try 5000 different iterations. Found a solution that moves 1007 regions; Going from a computed imbalance of 0.016471160309089496 to a new imbalance of 8.863329058851795E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.003312693299999688); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005124707431905383); MoveCostFunction : (multiplier=7.0, imbalance=0.11188888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.003312693299999688); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:53:29,672 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv571387681:82 , srv1317519828:83 , srv1434007054:83 , srv1506179239:83 , srv1647452802:83 , srv1656220437:83 , srv1956485304:83 , srv2054562125:83 , srv31762935:83 , srv560625111:83 , srv720417579:83 , srv1025931700:84 , srv1131201566:84 , srv1242736848:84 , srv2071710822:84 , srv349817455:84 , srv498600364:84 , srv517799152:84 , srv535861304:84 , srv584360703:84 , srv588763407:84 , srv762189627:84 , srv767909299:84 , srv844744509:84 , srv866375873:84 , srv914898664:84 , srv922005314:84 , srv977132599:84 , srv1017850351:85 , srv1100422842:85 , srv1263466912:85 , srv1306271367:85 , srv137534181:85 , srv1387536904:85 , srv1672118627:85 , srv2065572237:85 , srv2073345365:85 , srv235560416:85 , srv268833657:85 , srv269177671:85 , srv614892327:85 , srv65116946:85 , srv723107193:85 , srv798539124:85 , srv83905965:85 , srv894732152:85 , srv932424714:85 , srv957800658:85 , srv96723364:85 , srv969739394:85 , srv1150761673:86 , srv1175347504:86 , srv1221427148:86 , srv1223435392:86 , srv1235216844:86 , srv155106741:86 , srv1685820486:86 , srv1710015818:86 , srv1768375928:86 , srv1917604044:86 , srv2006150489:86 , srv2106961072:86 , srv279686493:86 , srv436279762:86 , srv552621924:86 , srv686307461:86 , srv738511247:86 , srv784923252:86 , srv831554074:86 , srv914613475:86 , srv969218489:86 , srv1053559404:87 , srv1250365830:87 , srv1298469133:87 , srv136877463:87 , srv153469746:87 , srv1631705580:87 , srv1744650430:87 , srv1755880675:87 , srv1832986359:87 , srv1971452360:87 , srv1973549731:87 , srv2095899647:87 , srv2120402559:87 , srv2120511941:87 , srv230623414:87 , srv477113741:87 , srv553298073:87 , srv579845024:87 , srv583181237:87 , srv770888256:87 , srv773187910:87 , srv851622295:87 , srv939352392:87 , srv105999388:88 , srv1406260689:88 , srv1692867915:88 , srv1727977942:88 , srv1908068330:88 , srv340660885:543 } 2023-10-16 11:53:29,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 11:53:29,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1263466912=13, srv1973549731=41, srv517799152=62, srv1025931700=1, srv137534181=18, srv436279762=59, srv773187910=81, srv1727977942=32, srv552621924=64, srv1956485304=39, srv831554074=84, srv2106961072=48, srv571387681=67, srv770888256=80, srv1406260689=20, srv1298469133=14, srv268833657=53, srv1908068330=37, srv1017850351=0, srv153469746=23, srv279686493=55, srv1053559404=2, srv1223435392=9, srv2120511941=50, srv1755880675=34, srv1306271367=15, srv579845024=68, srv1631705580=25, srv686307461=74, srv2071710822=45, srv583181237=69, srv230623414=51, srv477113741=60, srv2095899647=47, srv614892327=72, srv939352392=94, srv553298073=65, srv340660885=57, srv738511247=77, srv1506179239=22, srv1250365830=12, srv1175347504=7, srv155106741=24, srv535861304=63, srv498600364=61, srv969739394=98, srv1672118627=28, srv2065572237=44, srv957800658=95, srv844744509=86, srv584360703=70, srv932424714=93, srv560625111=66, srv588763407=71, srv1832986359=36, srv65116946=73, srv1100422842=4, srv1387536904=19, srv1917604044=38, srv767909299=79, srv1317519828=16, srv136877463=17, srv235560416=52, srv784923252=82, srv914898664=91, srv1692867915=30, srv2120402559=49, srv1685820486=29, srv1971452360=40, srv866375873=88, srv1434007054=21, srv96723364=96, srv1744650430=33, srv83905965=85, srv914613475=90, srv1647452802=26, srv1242736848=11, srv1221427148=8, srv1768375928=35, srv2054562125=43, srv2073345365=46, srv1235216844=10, srv31762935=56, srv977132599=99, srv2006150489=42, srv762189627=78, srv1131201566=5, srv723107193=76, srv1710015818=31, srv1656220437=27, srv269177671=54, srv105999388=3, srv349817455=58, srv894732152=89, srv1150761673=6, srv922005314=92, srv851622295=87, srv720417579=75, srv798539124=83, srv969218489=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 11:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 11:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 11:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 11:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 11:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 11:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 11:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 11:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 11:53:29,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 11:53:29,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-16 11:53:29,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-16 11:53:29,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-16 11:53:29,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-16 11:53:29,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-16 11:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-16 11:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-16 11:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-16 11:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-16 11:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-16 11:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-16 11:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-16 11:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-16 11:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-16 11:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-16 11:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-16 11:53:29,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-16 11:53:29,805 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.003312693299999688); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005124707431905383); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.003312693299999688); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:53:29,805 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-16 11:53:29,805 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.792817411776142E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.003312693299999688); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005124707431905383); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.003312693299999688); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-16 11:53:30,193 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 486 ms to try 5000 different iterations. Found a solution that moves 199 regions; Going from a computed imbalance of 8.792817411776142E-4 to a new imbalance of 4.318821432449826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.022111111111111113); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:53:30,194 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv517799152:82 , srv1017850351:83 , srv1053559404:83 , srv1692867915:83 , srv1755880675:83 , srv773187910:83 , srv844744509:83 , srv1025931700:84 , srv105999388:84 , srv1100422842:84 , srv1175347504:84 , srv1223435392:84 , srv1235216844:84 , srv1242736848:84 , srv1250365830:84 , srv1263466912:84 , srv1298469133:84 , srv1306271367:84 , srv1317519828:84 , srv136877463:84 , srv137534181:84 , srv1406260689:84 , srv1434007054:84 , srv1506179239:84 , srv153469746:84 , srv155106741:84 , srv1631705580:84 , srv1710015818:84 , srv1727977942:84 , srv1768375928:84 , srv1832986359:84 , srv1908068330:84 , srv1917604044:84 , srv1971452360:84 , srv1973549731:84 , srv2054562125:84 , srv2065572237:84 , srv2071710822:84 , srv2120402559:84 , srv268833657:84 , srv279686493:84 , srv436279762:84 , srv477113741:84 , srv498600364:84 , srv535861304:84 , srv553298073:84 , srv560625111:84 , srv571387681:84 , srv579845024:84 , srv583181237:84 , srv588763407:84 , srv65116946:84 , srv686307461:84 , srv720417579:84 , srv738511247:84 , srv767909299:84 , srv770888256:84 , srv831554074:84 , srv83905965:84 , srv851622295:84 , srv922005314:84 , srv939352392:84 , srv96723364:84 , srv969218489:84 , srv977132599:84 , srv1131201566:85 , srv1150761673:85 , srv1221427148:85 , srv1647452802:85 , srv1656220437:85 , srv1672118627:85 , srv1685820486:85 , srv1744650430:85 , srv1956485304:85 , srv2006150489:85 , srv2073345365:85 , srv2095899647:85 , srv2106961072:85 , srv2120511941:85 , srv230623414:85 , srv235560416:85 , srv269177671:85 , srv31762935:85 , srv349817455:85 , srv552621924:85 , srv584360703:85 , srv614892327:85 , srv723107193:85 , srv784923252:85 , srv798539124:85 , srv866375873:85 , srv894732152:85 , srv914898664:85 , srv932424714:85 , srv957800658:85 , srv969739394:85 , srv1387536904:86 , srv762189627:86 , srv914613475:86 , srv340660885:655 } 2023-10-16 11:53:30,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 11:53:30,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1263466912=13, srv1973549731=41, srv517799152=62, srv1025931700=1, srv137534181=18, srv436279762=59, srv773187910=81, srv1727977942=32, srv552621924=64, srv1956485304=39, srv831554074=84, srv2106961072=48, srv571387681=67, srv770888256=80, srv1406260689=20, srv1298469133=14, srv268833657=53, srv1908068330=37, srv1017850351=0, srv153469746=23, srv279686493=55, srv1053559404=2, srv1223435392=9, srv2120511941=50, srv1755880675=34, srv1306271367=15, srv579845024=68, srv1631705580=25, srv686307461=74, srv2071710822=45, srv583181237=69, srv230623414=51, srv477113741=60, srv2095899647=47, srv614892327=72, srv939352392=94, srv553298073=65, srv340660885=57, srv738511247=77, srv1506179239=22, srv1250365830=12, srv1175347504=7, srv155106741=24, srv535861304=63, srv498600364=61, srv969739394=98, srv1672118627=28, srv2065572237=44, srv957800658=95, srv844744509=86, srv584360703=70, srv932424714=93, srv560625111=66, srv588763407=71, srv1832986359=36, srv65116946=73, srv1100422842=4, srv1387536904=19, srv1917604044=38, srv767909299=79, srv1317519828=16, srv136877463=17, srv235560416=52, srv784923252=82, srv914898664=91, srv1692867915=30, srv2120402559=49, srv1685820486=29, srv1971452360=40, srv866375873=88, srv1434007054=21, srv96723364=96, srv1744650430=33, srv83905965=85, srv914613475=90, srv1647452802=26, srv1242736848=11, srv1221427148=8, srv1768375928=35, srv2054562125=43, srv2073345365=46, srv1235216844=10, srv31762935=56, srv977132599=99, srv2006150489=42, srv762189627=78, srv1131201566=5, srv723107193=76, srv1710015818=31, srv1656220437=27, srv269177671=54, srv105999388=3, srv349817455=58, srv894732152=89, srv1150761673=6, srv922005314=92, srv851622295=87, srv720417579=75, srv798539124=83, srv969218489=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 11:53:30,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 11:53:30,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 11:53:30,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 11:53:30,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 11:53:30,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 11:53:30,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 11:53:30,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 11:53:30,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 11:53:30,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 11:53:30,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 11:53:30,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 11:53:30,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 11:53:30,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 11:53:30,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 11:53:30,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 11:53:30,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 11:53:30,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 11:53:30,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 11:53:30,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 11:53:30,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 11:53:30,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 11:53:30,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 11:53:30,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 11:53:30,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 11:53:30,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 11:53:30,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 11:53:30,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 11:53:30,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 11:53:30,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 11:53:30,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 11:53:30,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-16 11:53:30,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-16 11:53:30,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-16 11:53:30,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-16 11:53:30,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-16 11:53:30,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-16 11:53:30,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-16 11:53:30,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-16 11:53:30,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-16 11:53:30,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-16 11:53:30,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-16 11:53:30,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-16 11:53:30,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-16 11:53:30,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-16 11:53:30,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-16 11:53:30,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-16 11:53:30,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-16 11:53:30,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-16 11:53:30,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-16 11:53:30,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-16 11:53:30,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-16 11:53:30,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-16 11:53:30,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-16 11:53:30,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-16 11:53:30,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-16 11:53:30,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-16 11:53:30,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-16 11:53:30,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-16 11:53:30,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-16 11:53:30,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-16 11:53:30,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-16 11:53:30,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-16 11:53:30,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-16 11:53:30,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-16 11:53:30,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-16 11:53:30,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-16 11:53:30,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-16 11:53:30,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-16 11:53:30,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-16 11:53:30,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-16 11:53:30,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-16 11:53:30,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-16 11:53:30,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-16 11:53:30,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-16 11:53:30,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-16 11:53:30,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-16 11:53:30,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-16 11:53:30,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-16 11:53:30,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-16 11:53:30,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-16 11:53:30,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-16 11:53:30,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-16 11:53:30,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-16 11:53:30,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-16 11:53:30,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-16 11:53:30,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-16 11:53:30,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-16 11:53:30,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-16 11:53:30,339 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:53:30,339 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-16 11:53:30,340 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.179478654216909E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-16 11:53:30,596 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 358 ms to try 5000 different iterations. Found a solution that moves 31 regions; Going from a computed imbalance of 4.179478654216909E-5 to a new imbalance of 7.604824139770872E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0034444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:53:30,596 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1387536904:83 , srv1744650430:83 , srv2095899647:83 , srv230623414:83 , srv235560416:83 , srv762189627:83 , srv914613475:83 , srv1017850351:84 , srv1025931700:84 , srv1053559404:84 , srv105999388:84 , srv1131201566:84 , srv1150761673:84 , srv1221427148:84 , srv1235216844:84 , srv1242736848:84 , srv1250365830:84 , srv1263466912:84 , srv1298469133:84 , srv1306271367:84 , srv1317519828:84 , srv136877463:84 , srv137534181:84 , srv1406260689:84 , srv1434007054:84 , srv1506179239:84 , srv153469746:84 , srv155106741:84 , srv1631705580:84 , srv1647452802:84 , srv1656220437:84 , srv1672118627:84 , srv1685820486:84 , srv1710015818:84 , srv1727977942:84 , srv1755880675:84 , srv1768375928:84 , srv1832986359:84 , srv1908068330:84 , srv1917604044:84 , srv1956485304:84 , srv1971452360:84 , srv1973549731:84 , srv2006150489:84 , srv2054562125:84 , srv2065572237:84 , srv2073345365:84 , srv2106961072:84 , srv2120402559:84 , srv2120511941:84 , srv268833657:84 , srv269177671:84 , srv279686493:84 , srv31762935:84 , srv349817455:84 , srv436279762:84 , srv477113741:84 , srv498600364:84 , srv535861304:84 , srv552621924:84 , srv553298073:84 , srv560625111:84 , srv571387681:84 , srv579845024:84 , srv583181237:84 , srv584360703:84 , srv588763407:84 , srv614892327:84 , srv65116946:84 , srv686307461:84 , srv720417579:84 , srv723107193:84 , srv738511247:84 , srv767909299:84 , srv770888256:84 , srv784923252:84 , srv798539124:84 , srv831554074:84 , srv83905965:84 , srv844744509:84 , srv851622295:84 , srv866375873:84 , srv894732152:84 , srv914898664:84 , srv922005314:84 , srv932424714:84 , srv939352392:84 , srv957800658:84 , srv96723364:84 , srv969218489:84 , srv969739394:84 , srv977132599:84 , srv1100422842:85 , srv1175347504:85 , srv1223435392:85 , srv1692867915:85 , srv2071710822:85 , srv517799152:85 , srv773187910:85 , srv340660885:684 } 2023-10-16 11:53:30,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 11:53:30,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1263466912=13, srv1973549731=41, srv517799152=62, srv1025931700=1, srv137534181=18, srv436279762=59, srv773187910=81, srv1727977942=32, srv552621924=64, srv1956485304=39, srv831554074=84, srv2106961072=48, srv571387681=67, srv770888256=80, srv1406260689=20, srv1298469133=14, srv268833657=53, srv1908068330=37, srv1017850351=0, srv153469746=23, srv279686493=55, srv1053559404=2, srv1223435392=9, srv2120511941=50, srv1755880675=34, srv1306271367=15, srv579845024=68, srv1631705580=25, srv686307461=74, srv2071710822=45, srv583181237=69, srv230623414=51, srv477113741=60, srv2095899647=47, srv614892327=72, srv939352392=94, srv553298073=65, srv340660885=57, srv738511247=77, srv1506179239=22, srv1250365830=12, srv1175347504=7, srv155106741=24, srv535861304=63, srv498600364=61, srv969739394=98, srv1672118627=28, srv2065572237=44, srv957800658=95, srv844744509=86, srv584360703=70, srv932424714=93, srv560625111=66, srv588763407=71, srv1832986359=36, srv65116946=73, srv1100422842=4, srv1387536904=19, srv1917604044=38, srv767909299=79, srv1317519828=16, srv136877463=17, srv235560416=52, srv784923252=82, srv914898664=91, srv1692867915=30, srv2120402559=49, srv1685820486=29, srv1971452360=40, srv866375873=88, srv1434007054=21, srv96723364=96, srv1744650430=33, srv83905965=85, srv914613475=90, srv1647452802=26, srv1242736848=11, srv1221427148=8, srv1768375928=35, srv2054562125=43, srv2073345365=46, srv1235216844=10, srv31762935=56, srv977132599=99, srv2006150489=42, srv762189627=78, srv1131201566=5, srv723107193=76, srv1710015818=31, srv1656220437=27, srv269177671=54, srv105999388=3, srv349817455=58, srv894732152=89, srv1150761673=6, srv922005314=92, srv851622295=87, srv720417579=75, srv798539124=83, srv969218489=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 11:53:30,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 11:53:30,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 11:53:30,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-16 11:53:30,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-16 11:53:30,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-16 11:53:30,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-16 11:53:30,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-16 11:53:30,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-16 11:53:30,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-16 11:53:30,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-16 11:53:30,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-16 11:53:30,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-16 11:53:30,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-16 11:53:30,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-16 11:53:30,725 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:53:30,726 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1387536904:83 , srv1744650430:83 , srv2095899647:83 , srv230623414:83 , srv235560416:83 , srv762189627:83 , srv914613475:83 , srv1017850351:84 , srv1025931700:84 , srv1053559404:84 , srv105999388:84 , srv1131201566:84 , srv1150761673:84 , srv1221427148:84 , srv1235216844:84 , srv1242736848:84 , srv1250365830:84 , srv1263466912:84 , srv1298469133:84 , srv1306271367:84 , srv1317519828:84 , srv136877463:84 , srv137534181:84 , srv1406260689:84 , srv1434007054:84 , srv1506179239:84 , srv153469746:84 , srv155106741:84 , srv1631705580:84 , srv1647452802:84 , srv1656220437:84 , srv1672118627:84 , srv1685820486:84 , srv1710015818:84 , srv1727977942:84 , srv1755880675:84 , srv1768375928:84 , srv1832986359:84 , srv1908068330:84 , srv1917604044:84 , srv1956485304:84 , srv1971452360:84 , srv1973549731:84 , srv2006150489:84 , srv2054562125:84 , srv2065572237:84 , srv2073345365:84 , srv2106961072:84 , srv2120402559:84 , srv2120511941:84 , srv268833657:84 , srv269177671:84 , srv279686493:84 , srv31762935:84 , srv349817455:84 , srv436279762:84 , srv477113741:84 , srv498600364:84 , srv535861304:84 , srv552621924:84 , srv553298073:84 , srv560625111:84 , srv571387681:84 , srv579845024:84 , srv583181237:84 , srv584360703:84 , srv588763407:84 , srv614892327:84 , srv65116946:84 , srv686307461:84 , srv720417579:84 , srv723107193:84 , srv738511247:84 , srv767909299:84 , srv770888256:84 , srv784923252:84 , srv798539124:84 , srv831554074:84 , srv83905965:84 , srv844744509:84 , srv851622295:84 , srv866375873:84 , srv894732152:84 , srv914898664:84 , srv922005314:84 , srv932424714:84 , srv939352392:84 , srv957800658:84 , srv96723364:84 , srv969218489:84 , srv969739394:84 , srv977132599:84 , srv1100422842:85 , srv1175347504:85 , srv1223435392:85 , srv1692867915:85 , srv2071710822:85 , srv517799152:85 , srv773187910:85 , srv340660885:684 } 2023-10-16 11:53:30,752 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=389 (was 336) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=5514 (was 6029) 2023-10-16 11:53:30,763 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=389, ProcessCount=170, AvailableMemoryMB=5514 2023-10-16 11:53:30,764 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-16 11:53:30,764 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-16 11:53:30,773 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011814377:84 , srv1071831659:84 , srv107945542:84 , srv1104585565:84 , srv1159068518:84 , srv1214110188:84 , srv1225676841:84 , srv1262703804:84 , srv1609940153:84 , srv1753823606:84 , srv1863617446:84 , srv1867067375:84 , srv1879485742:84 , srv2048064942:84 , srv2060397138:84 , srv2079752603:84 , srv2094745452:84 , srv2094877367:84 , srv336195060:84 , srv346450825:84 , srv43111537:84 , srv513132558:84 , srv580924593:84 , srv594257798:84 , srv619333929:84 , srv77698018:84 , srv88805586:84 , srv950867531:84 , srv998618746:84 , srv566344750:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-16 11:53:30,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 11:53:30,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1159068518=4, srv1011814377=0, srv1867067375=11, srv346450825=19, srv1262703804=7, srv2094877367=17, srv1609940153=8, srv1863617446=10, srv1214110188=5, srv77698018=26, srv950867531=28, srv1879485742=12, srv594257798=24, srv998618746=29, srv1753823606=9, srv1071831659=1, srv107945542=2, srv336195060=18, srv2048064942=13, srv1225676841=6, srv43111537=20, srv2060397138=14, srv88805586=27, srv566344750=22, srv513132558=21, srv619333929=25, srv2094745452=16, srv2079752603=15, srv1104585565=3, srv580924593=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 11:53:30,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 11:53:30,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 11:53:30,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 11:53:30,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 11:53:30,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 11:53:30,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 11:53:30,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 11:53:30,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 11:53:30,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 11:53:30,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 11:53:30,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 11:53:30,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 11:53:30,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 11:53:30,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 11:53:30,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 11:53:30,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 11:53:30,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 11:53:30,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-16 11:53:30,814 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.124082470080078, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:53:30,814 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912584023075837, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.124082470080078, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-16 11:55:30,795 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5551490 different iterations. Found a solution that moves 1865 regions; Going from a computed imbalance of 0.9912584023075837 to a new imbalance of 4.353003038599517E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6907407407407408, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:55:30,802 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1011814377:90 , srv1071831659:90 , srv107945542:90 , srv1104585565:90 , srv1159068518:90 , srv1214110188:90 , srv1225676841:90 , srv1262703804:90 , srv1609940153:90 , srv1753823606:90 , srv1863617446:90 , srv1867067375:90 , srv1879485742:90 , srv2048064942:90 , srv2060397138:90 , srv2079752603:90 , srv2094745452:90 , srv2094877367:90 , srv336195060:90 , srv346450825:90 , srv43111537:90 , srv513132558:90 , srv566344750:90 , srv580924593:90 , srv594257798:90 , srv619333929:90 , srv77698018:90 , srv88805586:90 , srv950867531:90 , srv998618746:90 } 2023-10-16 11:55:30,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-16 11:55:30,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1159068518=4, srv1011814377=0, srv1867067375=11, srv346450825=19, srv1262703804=7, srv2094877367=17, srv1609940153=8, srv1863617446=10, srv1214110188=5, srv77698018=26, srv950867531=28, srv1879485742=12, srv594257798=24, srv998618746=29, srv1753823606=9, srv1071831659=1, srv107945542=2, srv336195060=18, srv2048064942=13, srv1225676841=6, srv43111537=20, srv2060397138=14, srv88805586=27, srv566344750=22, srv513132558=21, srv619333929=25, srv2094745452=16, srv2079752603=15, srv1104585565=3, srv580924593=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-16 11:55:30,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-16 11:55:30,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-16 11:55:30,827 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-16 11:55:30,827 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1011814377:90 , srv1071831659:90 , srv107945542:90 , srv1104585565:90 , srv1159068518:90 , srv1214110188:90 , srv1225676841:90 , srv1262703804:90 , srv1609940153:90 , srv1753823606:90 , srv1863617446:90 , srv1867067375:90 , srv1879485742:90 , srv2048064942:90 , srv2060397138:90 , srv2079752603:90 , srv2094745452:90 , srv2094877367:90 , srv336195060:90 , srv346450825:90 , srv43111537:90 , srv513132558:90 , srv566344750:90 , srv580924593:90 , srv594257798:90 , srv619333929:90 , srv77698018:90 , srv88805586:90 , srv950867531:90 , srv998618746:90 } 2023-10-16 11:55:30,843 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=225 (was 389), ProcessCount=173 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=5438 (was 5514)