2024-01-25 12:14:38,833 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-01-25 12:14:39,149 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-25 12:14:39,267 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-25 12:14:39,340 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-25 12:14:39,347 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-25 12:14:39,362 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=275, ProcessCount=170, AvailableMemoryMB=9995 2024-01-25 12:14:39,365 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-25 12:14:39,365 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-25 12:14:39,513 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013832330:84 , srv1027645286:84 , srv1046375347:84 , srv1067442915:84 , srv1079958520:84 , srv1088956850:84 , srv1107580219:84 , srv1140385440:84 , srv115983992:84 , srv1159944459:84 , srv1170794044:84 , srv1184451494:84 , srv120088990:84 , srv1228959325:84 , srv1229734258:84 , srv124870115:84 , srv1273313777:84 , srv1287897741:84 , srv1303793754:84 , srv1327303540:84 , srv1339099818:84 , srv1359195055:84 , srv1405242608:84 , srv1411979577:84 , srv1444009347:84 , srv1453544184:84 , srv1495253164:84 , srv1517104167:84 , srv1559787211:84 , srv1571474523:84 , srv1617121609:84 , srv1650362002:84 , srv1663914481:84 , srv1670637447:84 , srv167562419:84 , srv1675758893:84 , srv1685388018:84 , srv1710491515:84 , srv1716775928:84 , srv1722550716:84 , srv1725901793:84 , srv1731631592:84 , srv1733118208:84 , srv1758317488:84 , srv1759926323:84 , srv1819025104:84 , srv1827581363:84 , srv1837756566:84 , srv1876507878:84 , srv1880036899:84 , srv1889481870:84 , srv1895425474:84 , srv1906073214:84 , srv1960774649:84 , srv201777660:84 , srv2018184644:84 , srv2027905388:84 , srv2035309810:84 , srv2038035955:84 , srv205341357:84 , srv2054453124:84 , srv213397404:84 , srv22155102:84 , srv268584938:84 , srv274291077:84 , srv275352161:84 , srv291158136:84 , srv310220447:84 , srv314072677:84 , srv335434584:84 , srv354048751:84 , srv365240527:84 , srv368379677:84 , srv397261457:84 , srv416857571:84 , srv423756036:84 , srv424302764:84 , srv459381803:84 , srv466877395:84 , srv536285751:84 , srv554761947:84 , srv589737013:84 , srv639766334:84 , srv641665145:84 , srv649483775:84 , srv662339871:84 , srv698759558:84 , srv702487129:84 , srv706178541:84 , srv710428285:84 , srv740446547:84 , srv744611412:84 , srv785425627:84 , srv856365713:84 , srv907153600:84 , srv910240561:84 , srv966329852:84 , srv982240109:84 , srv985256531:84 , srv2102715508:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-01-25 12:14:39,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 12:14:39,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv423756036=76, srv589737013=82, srv1273313777=16, srv115983992=8, srv1229734258=14, srv1827581363=46, srv1876507878=48, srv966329852=97, srv1339099818=20, srv1733118208=42, srv1517104167=27, srv1559787211=28, srv1159944459=9, srv397261457=74, srv1722550716=39, srv662339871=86, srv1027645286=1, srv167562419=34, srv201777660=54, srv785425627=93, srv354048751=71, srv1107580219=6, srv1303793754=18, srv205341357=59, srv124870115=15, srv1716775928=38, srv1758317488=43, srv310220447=68, srv641665145=84, srv1960774649=53, srv1906073214=52, srv740446547=91, srv2018184644=55, srv368379677=73, srv2054453124=60, srv1327303540=19, srv1013832330=0, srv1088956850=5, srv1170794044=10, srv1067442915=3, srv2035309810=57, srv459381803=78, srv1359195055=21, srv120088990=12, srv335434584=70, srv466877395=79, srv554761947=81, srv1617121609=30, srv22155102=63, srv1444009347=24, srv365240527=72, srv1228959325=13, srv1675758893=35, srv2038035955=58, srv1411979577=23, srv1759926323=44, srv268584938=64, srv649483775=85, srv1725901793=40, srv1895425474=51, srv985256531=99, srv910240561=96, srv702487129=88, srv982240109=98, srv416857571=75, srv856365713=94, srv424302764=77, srv1650362002=31, srv1880036899=49, srv1184451494=11, srv1685388018=36, srv1287897741=17, srv1663914481=32, srv1731631592=41, srv1670637447=33, srv1495253164=26, srv1405242608=22, srv2027905388=56, srv1819025104=45, srv314072677=69, srv274291077=65, srv1140385440=7, srv1889481870=50, srv639766334=83, srv907153600=95, srv706178541=89, srv291158136=67, srv1837756566=47, srv1571474523=29, srv2102715508=61, srv536285751=80, srv698759558=87, srv1710491515=37, srv213397404=62, srv275352161=66, srv1046375347=2, srv1453544184=25, srv710428285=90, srv744611412=92, srv1079958520=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 12:14:39,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 12:14:39,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 12:14:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 12:14:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 12:14:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 12:14:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 12:14:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 12:14:39,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 12:14:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 12:14:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 12:14:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 12:14:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 12:14:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 12:14:39,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 12:14:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 12:14:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 12:14:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 12:14:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 12:14:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 12:14:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 12:14:39,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 12:14:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 12:14:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 12:14:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 12:14:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 12:14:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 12:14:39,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 12:14:39,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 12:14:39,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 12:14:39,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 12:14:39,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-25 12:14:39,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-25 12:14:39,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-25 12:14:39,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-25 12:14:39,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-25 12:14:39,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-25 12:14:39,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-25 12:14:39,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-25 12:14:39,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-25 12:14:39,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-25 12:14:39,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-25 12:14:39,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-25 12:14:39,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-25 12:14:39,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-25 12:14:39,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-25 12:14:39,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-25 12:14:39,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-25 12:14:39,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-25 12:14:39,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-25 12:14:39,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-25 12:14:39,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-25 12:14:39,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-25 12:14:39,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-25 12:14:39,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-25 12:14:39,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-25 12:14:39,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-25 12:14:39,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-25 12:14:39,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-25 12:14:39,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-25 12:14:39,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-25 12:14:39,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-25 12:14:39,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-25 12:14:39,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-25 12:14:39,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-25 12:14:39,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-25 12:14:39,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-25 12:14:39,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-25 12:14:39,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-25 12:14:39,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-25 12:14:39,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-25 12:14:39,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-25 12:14:39,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-25 12:14:39,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-25 12:14:39,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-25 12:14:39,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-25 12:14:39,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-25 12:14:39,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-25 12:14:39,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-25 12:14:39,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-25 12:14:39,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-25 12:14:39,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-25 12:14:39,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-25 12:14:39,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-25 12:14:39,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-25 12:14:39,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-25 12:14:39,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-25 12:14:39,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-25 12:14:39,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-25 12:14:39,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-25 12:14:39,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-25 12:14:39,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-25 12:14:39,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-25 12:14:39,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-25 12:14:39,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-25 12:14:39,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-25 12:14:39,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-25 12:14:39,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-25 12:14:39,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-25 12:14:39,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-25 12:14:39,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-25 12:14:39,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-25 12:14:39,926 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:14:39,927 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-25 12:14:39,927 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-25 12:14:40,379 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 713 ms to try 5000 different iterations. Found a solution that moves 4693 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18434025977201646. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04004798219529858, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05677450622134806, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5214444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04004798219529858, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17958333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:14:40,396 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1670637447:81 , srv1876507878:82 , srv310220447:82 , srv1067442915:83 , srv115983992:83 , srv1617121609:83 , srv1716775928:83 , srv1725901793:83 , srv365240527:83 , srv466877395:83 , srv740446547:83 , srv910240561:83 , srv985256531:83 , srv1184451494:84 , srv1287897741:84 , srv167562419:84 , srv1733118208:84 , srv1758317488:84 , srv268584938:84 , srv416857571:84 , srv639766334:84 , srv1013832330:85 , srv1159944459:85 , srv1228959325:85 , srv1229734258:85 , srv1339099818:85 , srv1359195055:85 , srv1495253164:85 , srv1731631592:85 , srv1889481870:85 , srv2054453124:85 , srv275352161:85 , srv354048751:85 , srv536285751:85 , srv706178541:85 , srv1046375347:86 , srv1140385440:86 , srv120088990:86 , srv124870115:86 , srv1273313777:86 , srv1327303540:86 , srv1559787211:86 , srv1675758893:86 , srv1759926323:86 , srv1880036899:86 , srv1960774649:86 , srv2018184644:86 , srv213397404:86 , srv22155102:86 , srv649483775:86 , srv662339871:86 , srv698759558:86 , srv785425627:86 , srv856365713:86 , srv907153600:86 , srv966329852:86 , srv1027645286:87 , srv1444009347:87 , srv1722550716:87 , srv1837756566:87 , srv1895425474:87 , srv201777660:87 , srv2038035955:87 , srv205341357:87 , srv314072677:87 , srv459381803:87 , srv589737013:87 , srv641665145:87 , srv744611412:87 , srv1170794044:88 , srv1303793754:88 , srv1411979577:88 , srv1517104167:88 , srv1571474523:88 , srv1650362002:88 , srv1685388018:88 , srv1710491515:88 , srv2027905388:88 , srv274291077:88 , srv423756036:88 , srv554761947:88 , srv982240109:88 , srv1453544184:89 , srv1663914481:89 , srv1819025104:89 , srv291158136:89 , srv424302764:89 , srv1088956850:90 , srv1107580219:90 , srv1906073214:90 , srv368379677:90 , srv397261457:90 , srv710428285:90 , srv1079958520:91 , srv1405242608:91 , srv1827581363:91 , srv335434584:91 , srv702487129:93 , srv2035309810:94 , srv2102715508:446 } 2024-01-25 12:14:40,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 12:14:40,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv423756036=76, srv589737013=82, srv1273313777=16, srv115983992=8, srv1229734258=14, srv1827581363=46, srv1876507878=48, srv966329852=97, srv1339099818=20, srv1733118208=42, srv1517104167=27, srv1559787211=28, srv1159944459=9, srv397261457=74, srv1722550716=39, srv662339871=86, srv1027645286=1, srv167562419=34, srv201777660=54, srv785425627=93, srv354048751=71, srv1107580219=6, srv1303793754=18, srv205341357=59, srv124870115=15, srv1716775928=38, srv1758317488=43, srv310220447=68, srv641665145=84, srv1960774649=53, srv1906073214=52, srv740446547=91, srv2018184644=55, srv368379677=73, srv2054453124=60, srv1327303540=19, srv1013832330=0, srv1088956850=5, srv1170794044=10, srv1067442915=3, srv2035309810=57, srv459381803=78, srv1359195055=21, srv120088990=12, srv335434584=70, srv466877395=79, srv554761947=81, srv1617121609=30, srv22155102=63, srv1444009347=24, srv365240527=72, srv1228959325=13, srv1675758893=35, srv2038035955=58, srv1411979577=23, srv1759926323=44, srv268584938=64, srv649483775=85, srv1725901793=40, srv1895425474=51, srv985256531=99, srv910240561=96, srv702487129=88, srv982240109=98, srv416857571=75, srv856365713=94, srv424302764=77, srv1650362002=31, srv1880036899=49, srv1184451494=11, srv1685388018=36, srv1287897741=17, srv1663914481=32, srv1731631592=41, srv1670637447=33, srv1495253164=26, srv1405242608=22, srv2027905388=56, srv1819025104=45, srv314072677=69, srv274291077=65, srv1140385440=7, srv1889481870=50, srv639766334=83, srv907153600=95, srv706178541=89, srv291158136=67, srv1837756566=47, srv1571474523=29, srv2102715508=61, srv536285751=80, srv698759558=87, srv1710491515=37, srv213397404=62, srv275352161=66, srv1046375347=2, srv1453544184=25, srv710428285=90, srv744611412=92, srv1079958520=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 12:14:40,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 12:14:40,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 12:14:40,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 12:14:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 12:14:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 12:14:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 12:14:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 12:14:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 12:14:40,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 12:14:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 12:14:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 12:14:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 12:14:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 12:14:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 12:14:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 12:14:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 12:14:40,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 12:14:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 12:14:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 12:14:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 12:14:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 12:14:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 12:14:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 12:14:40,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 12:14:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 12:14:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 12:14:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 12:14:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 12:14:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 12:14:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 12:14:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-25 12:14:40,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-25 12:14:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-25 12:14:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-25 12:14:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-25 12:14:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-25 12:14:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-25 12:14:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-25 12:14:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-25 12:14:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-25 12:14:40,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-25 12:14:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-25 12:14:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-25 12:14:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-25 12:14:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-25 12:14:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-25 12:14:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-25 12:14:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-25 12:14:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-25 12:14:40,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-25 12:14:40,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-25 12:14:40,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-25 12:14:40,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-25 12:14:40,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-25 12:14:40,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-25 12:14:40,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-25 12:14:40,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-25 12:14:40,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-25 12:14:40,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-25 12:14:40,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-25 12:14:40,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-25 12:14:40,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-25 12:14:40,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-25 12:14:40,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-25 12:14:40,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-25 12:14:40,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-25 12:14:40,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-25 12:14:40,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-25 12:14:40,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-25 12:14:40,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-25 12:14:40,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-25 12:14:40,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-25 12:14:40,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-25 12:14:40,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-25 12:14:40,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-25 12:14:40,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-25 12:14:40,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-25 12:14:40,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-25 12:14:40,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-25 12:14:40,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-25 12:14:40,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-25 12:14:40,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-25 12:14:40,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-25 12:14:40,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-25 12:14:40,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-25 12:14:40,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-25 12:14:40,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-25 12:14:40,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-25 12:14:40,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-25 12:14:40,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-25 12:14:40,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-25 12:14:40,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-25 12:14:40,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-25 12:14:40,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-25 12:14:40,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-25 12:14:40,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-25 12:14:40,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-25 12:14:40,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-25 12:14:40,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-25 12:14:40,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-25 12:14:40,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-25 12:14:40,611 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04004798219529858, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05677450622134806, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04004798219529858, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17958333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:14:40,611 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-25 12:14:40,612 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18430739868366236, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04004798219529858, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05677450622134806, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04004798219529858, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17958333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-25 12:14:40,923 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 453 ms to try 5000 different iterations. Found a solution that moves 2601 regions; Going from a computed imbalance of 0.18430739868366236 to a new imbalance of 0.015679769909069018. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02373367213300172); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03554924186367313, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.289, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02373367213300172); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:14:40,935 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2035309810:77 , srv1827581363:80 , srv1906073214:80 , srv702487129:80 , srv710428285:80 , srv1088956850:81 , srv1107580219:81 , srv335434584:81 , srv368379677:81 , srv423756036:82 , srv1046375347:83 , srv1079958520:83 , srv1170794044:83 , srv1405242608:83 , srv1517104167:83 , srv1685388018:83 , srv1710491515:83 , srv201777660:83 , srv291158136:83 , srv1303793754:84 , srv1411979577:84 , srv1453544184:84 , srv1650362002:84 , srv1663914481:84 , srv1759926323:84 , srv1889481870:84 , srv1895425474:84 , srv205341357:84 , srv314072677:84 , srv424302764:84 , srv554761947:84 , srv641665145:84 , srv966329852:84 , srv1027645286:85 , srv1140385440:85 , srv1184451494:85 , srv1229734258:85 , srv1273313777:85 , srv1559787211:85 , srv1571474523:85 , srv1675758893:85 , srv1880036899:85 , srv2027905388:85 , srv213397404:85 , srv354048751:85 , srv397261457:85 , srv459381803:85 , srv589737013:85 , srv649483775:85 , srv662339871:85 , srv1287897741:86 , srv1327303540:86 , srv1339099818:86 , srv1444009347:86 , srv1722550716:86 , srv1819025104:86 , srv2018184644:86 , srv2038035955:86 , srv2054453124:86 , srv274291077:86 , srv536285751:86 , srv698759558:86 , srv706178541:86 , srv744611412:86 , srv982240109:86 , srv1013832330:87 , srv1067442915:87 , srv1159944459:87 , srv120088990:87 , srv124870115:87 , srv1716775928:87 , srv1731631592:87 , srv1758317488:87 , srv1837756566:87 , srv1960774649:87 , srv740446547:87 , srv1359195055:88 , srv167562419:88 , srv1733118208:88 , srv22155102:88 , srv310220447:88 , srv785425627:88 , srv856365713:88 , srv907153600:88 , srv910240561:88 , srv115983992:89 , srv1228959325:89 , srv1495253164:89 , srv1670637447:89 , srv1725901793:89 , srv275352161:89 , srv365240527:89 , srv466877395:89 , srv639766334:89 , srv1617121609:90 , srv1876507878:90 , srv268584938:90 , srv416857571:90 , srv985256531:90 , srv2102715508:539 } 2024-01-25 12:14:40,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 12:14:40,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv423756036=76, srv589737013=82, srv1273313777=16, srv115983992=8, srv1229734258=14, srv1827581363=46, srv1876507878=48, srv966329852=97, srv1339099818=20, srv1733118208=42, srv1517104167=27, srv1559787211=28, srv1159944459=9, srv397261457=74, srv1722550716=39, srv662339871=86, srv1027645286=1, srv167562419=34, srv201777660=54, srv785425627=93, srv354048751=71, srv1107580219=6, srv1303793754=18, srv205341357=59, srv124870115=15, srv1716775928=38, srv1758317488=43, srv310220447=68, srv641665145=84, srv1960774649=53, srv1906073214=52, srv740446547=91, srv2018184644=55, srv368379677=73, srv2054453124=60, srv1327303540=19, srv1013832330=0, srv1088956850=5, srv1170794044=10, srv1067442915=3, srv2035309810=57, srv459381803=78, srv1359195055=21, srv120088990=12, srv335434584=70, srv466877395=79, srv554761947=81, srv1617121609=30, srv22155102=63, srv1444009347=24, srv365240527=72, srv1228959325=13, srv1675758893=35, srv2038035955=58, srv1411979577=23, srv1759926323=44, srv268584938=64, srv649483775=85, srv1725901793=40, srv1895425474=51, srv985256531=99, srv910240561=96, srv702487129=88, srv982240109=98, srv416857571=75, srv856365713=94, srv424302764=77, srv1650362002=31, srv1880036899=49, srv1184451494=11, srv1685388018=36, srv1287897741=17, srv1663914481=32, srv1731631592=41, srv1670637447=33, srv1495253164=26, srv1405242608=22, srv2027905388=56, srv1819025104=45, srv314072677=69, srv274291077=65, srv1140385440=7, srv1889481870=50, srv639766334=83, srv907153600=95, srv706178541=89, srv291158136=67, srv1837756566=47, srv1571474523=29, srv2102715508=61, srv536285751=80, srv698759558=87, srv1710491515=37, srv213397404=62, srv275352161=66, srv1046375347=2, srv1453544184=25, srv710428285=90, srv744611412=92, srv1079958520=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 12:14:40,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 12:14:40,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 12:14:40,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 12:14:40,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 12:14:40,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 12:14:40,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 12:14:40,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 12:14:40,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 12:14:40,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 12:14:40,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 12:14:40,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 12:14:40,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 12:14:40,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 12:14:40,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 12:14:40,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 12:14:40,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 12:14:40,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 12:14:40,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 12:14:40,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 12:14:40,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 12:14:40,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 12:14:40,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 12:14:40,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 12:14:40,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 12:14:40,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 12:14:40,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 12:14:40,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 12:14:40,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 12:14:40,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 12:14:40,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 12:14:40,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-25 12:14:40,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-25 12:14:40,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-25 12:14:40,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-25 12:14:40,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-25 12:14:40,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-25 12:14:40,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-25 12:14:40,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-25 12:14:40,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-25 12:14:40,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-25 12:14:40,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-25 12:14:40,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-25 12:14:40,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-25 12:14:40,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-25 12:14:40,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-25 12:14:40,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-25 12:14:40,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-25 12:14:40,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-25 12:14:40,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-25 12:14:40,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-25 12:14:40,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-25 12:14:40,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-25 12:14:40,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-25 12:14:40,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-25 12:14:40,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-25 12:14:40,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-25 12:14:40,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-25 12:14:40,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-25 12:14:40,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-25 12:14:40,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-25 12:14:40,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-25 12:14:40,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-25 12:14:40,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-25 12:14:40,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-25 12:14:40,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-25 12:14:40,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-25 12:14:40,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-25 12:14:40,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-25 12:14:40,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-25 12:14:40,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-25 12:14:40,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-25 12:14:40,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-25 12:14:40,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-25 12:14:40,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-25 12:14:40,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-25 12:14:40,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-25 12:14:40,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-25 12:14:40,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-25 12:14:40,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-25 12:14:40,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-25 12:14:40,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-25 12:14:40,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-25 12:14:40,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-25 12:14:40,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-25 12:14:40,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-25 12:14:40,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-25 12:14:40,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-25 12:14:40,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-25 12:14:40,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-25 12:14:40,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-25 12:14:40,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-25 12:14:40,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-25 12:14:40,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-25 12:14:40,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-25 12:14:40,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-25 12:14:40,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-25 12:14:40,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-25 12:14:40,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-25 12:14:40,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-25 12:14:40,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-25 12:14:40,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-25 12:14:41,124 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02373367213300172); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03554924186367313, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02373367213300172); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:14:41,124 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-25 12:14:41,124 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0156615573178035, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02373367213300172); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03554924186367313, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02373367213300172); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-25 12:14:41,390 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 385 ms to try 5000 different iterations. Found a solution that moves 952 regions; Going from a computed imbalance of 0.0156615573178035 to a new imbalance of 7.891726542454146E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007106690545187014); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007899480725663495); MoveCostFunction : (multiplier=7.0, imbalance=0.10577777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007106690545187014); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:14:41,392 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1228959325:82 , srv1819025104:82 , srv397261457:82 , srv1495253164:83 , srv1837756566:83 , srv22155102:83 , srv274291077:83 , srv982240109:83 , srv1027645286:84 , srv1079958520:84 , srv120088990:84 , srv124870115:84 , srv1359195055:84 , srv1405242608:84 , srv1453544184:84 , srv1663914481:84 , srv1722550716:84 , srv1960774649:84 , srv275352161:84 , srv416857571:84 , srv424302764:84 , srv785425627:84 , srv856365713:84 , srv907153600:84 , srv985256531:84 , srv1159944459:85 , srv1327303540:85 , srv1411979577:85 , srv1571474523:85 , srv1617121609:85 , srv1650362002:85 , srv167562419:85 , srv1725901793:85 , srv1731631592:85 , srv1733118208:85 , srv1876507878:85 , srv2038035955:85 , srv268584938:85 , srv291158136:85 , srv335434584:85 , srv365240527:85 , srv459381803:85 , srv554761947:85 , srv639766334:85 , srv702487129:85 , srv744611412:85 , srv1013832330:86 , srv1088956850:86 , srv1140385440:86 , srv115983992:86 , srv1170794044:86 , srv1229734258:86 , srv1273313777:86 , srv1303793754:86 , srv1339099818:86 , srv1444009347:86 , srv1670637447:86 , srv1675758893:86 , srv1710491515:86 , srv1880036899:86 , srv1895425474:86 , srv2018184644:86 , srv2027905388:86 , srv2035309810:86 , srv2054453124:86 , srv213397404:86 , srv314072677:86 , srv368379677:86 , srv466877395:86 , srv536285751:86 , srv589737013:86 , srv662339871:86 , srv698759558:86 , srv706178541:86 , srv966329852:86 , srv1046375347:87 , srv1067442915:87 , srv1107580219:87 , srv1287897741:87 , srv1517104167:87 , srv1559787211:87 , srv1685388018:87 , srv1758317488:87 , srv1759926323:87 , srv1827581363:87 , srv1906073214:87 , srv205341357:87 , srv423756036:87 , srv641665145:87 , srv649483775:87 , srv710428285:87 , srv910240561:87 , srv1184451494:88 , srv1716775928:88 , srv201777660:88 , srv310220447:88 , srv354048751:88 , srv740446547:88 , srv1889481870:89 , srv2102715508:536 } 2024-01-25 12:14:41,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 12:14:41,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv423756036=76, srv589737013=82, srv1273313777=16, srv115983992=8, srv1229734258=14, srv1827581363=46, srv1876507878=48, srv966329852=97, srv1339099818=20, srv1733118208=42, srv1517104167=27, srv1559787211=28, srv1159944459=9, srv397261457=74, srv1722550716=39, srv662339871=86, srv1027645286=1, srv167562419=34, srv201777660=54, srv785425627=93, srv354048751=71, srv1107580219=6, srv1303793754=18, srv205341357=59, srv124870115=15, srv1716775928=38, srv1758317488=43, srv310220447=68, srv641665145=84, srv1960774649=53, srv1906073214=52, srv740446547=91, srv2018184644=55, srv368379677=73, srv2054453124=60, srv1327303540=19, srv1013832330=0, srv1088956850=5, srv1170794044=10, srv1067442915=3, srv2035309810=57, srv459381803=78, srv1359195055=21, srv120088990=12, srv335434584=70, srv466877395=79, srv554761947=81, srv1617121609=30, srv22155102=63, srv1444009347=24, srv365240527=72, srv1228959325=13, srv1675758893=35, srv2038035955=58, srv1411979577=23, srv1759926323=44, srv268584938=64, srv649483775=85, srv1725901793=40, srv1895425474=51, srv985256531=99, srv910240561=96, srv702487129=88, srv982240109=98, srv416857571=75, srv856365713=94, srv424302764=77, srv1650362002=31, srv1880036899=49, srv1184451494=11, srv1685388018=36, srv1287897741=17, srv1663914481=32, srv1731631592=41, srv1670637447=33, srv1495253164=26, srv1405242608=22, srv2027905388=56, srv1819025104=45, srv314072677=69, srv274291077=65, srv1140385440=7, srv1889481870=50, srv639766334=83, srv907153600=95, srv706178541=89, srv291158136=67, srv1837756566=47, srv1571474523=29, srv2102715508=61, srv536285751=80, srv698759558=87, srv1710491515=37, srv213397404=62, srv275352161=66, srv1046375347=2, srv1453544184=25, srv710428285=90, srv744611412=92, srv1079958520=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 12:14:41,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 12:14:41,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 12:14:41,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 12:14:41,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 12:14:41,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 12:14:41,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-25 12:14:41,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-25 12:14:41,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-25 12:14:41,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-25 12:14:41,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-25 12:14:41,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-25 12:14:41,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-25 12:14:41,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-25 12:14:41,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-25 12:14:41,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-25 12:14:41,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-25 12:14:41,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-25 12:14:41,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-25 12:14:41,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-25 12:14:41,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-25 12:14:41,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-25 12:14:41,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-25 12:14:41,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-25 12:14:41,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-25 12:14:41,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-25 12:14:41,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-25 12:14:41,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-25 12:14:41,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-25 12:14:41,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-25 12:14:41,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-25 12:14:41,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-25 12:14:41,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-25 12:14:41,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-25 12:14:41,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-25 12:14:41,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-25 12:14:41,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-25 12:14:41,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-25 12:14:41,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-25 12:14:41,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-25 12:14:41,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-25 12:14:41,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-25 12:14:41,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-25 12:14:41,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-25 12:14:41,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-25 12:14:41,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-25 12:14:41,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-25 12:14:41,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-25 12:14:41,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-25 12:14:41,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-25 12:14:41,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-25 12:14:41,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-25 12:14:41,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-25 12:14:41,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-25 12:14:41,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-25 12:14:41,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-25 12:14:41,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-25 12:14:41,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-25 12:14:41,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-25 12:14:41,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-25 12:14:41,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-25 12:14:41,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-25 12:14:41,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-25 12:14:41,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-25 12:14:41,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-25 12:14:41,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-25 12:14:41,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-25 12:14:41,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-25 12:14:41,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-25 12:14:41,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-25 12:14:41,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-25 12:14:41,530 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007106690545187014); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007899480725663495); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007106690545187014); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:14:41,531 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-25 12:14:41,531 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.825066077691464E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007106690545187014); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007899480725663495); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007106690545187014); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-25 12:14:41,986 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 554 ms to try 5000 different iterations. Found a solution that moves 283 regions; Going from a computed imbalance of 7.825066077691464E-4 to a new imbalance of 4.392184568642931E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.03144444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:14:41,987 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2027905388:83 , srv649483775:83 , srv856365713:83 , srv1013832330:84 , srv1079958520:84 , srv1107580219:84 , srv115983992:84 , srv1303793754:84 , srv1405242608:84 , srv1444009347:84 , srv1453544184:84 , srv1517104167:84 , srv1571474523:84 , srv1685388018:84 , srv1716775928:84 , srv1731631592:84 , srv1758317488:84 , srv1827581363:84 , srv1889481870:84 , srv201777660:84 , srv2018184644:84 , srv2038035955:84 , srv268584938:84 , srv275352161:84 , srv310220447:84 , srv354048751:84 , srv365240527:84 , srv368379677:84 , srv416857571:84 , srv466877395:84 , srv536285751:84 , srv589737013:84 , srv698759558:84 , srv702487129:84 , srv740446547:84 , srv785425627:84 , srv907153600:84 , srv910240561:84 , srv1027645286:85 , srv1046375347:85 , srv1067442915:85 , srv1088956850:85 , srv1140385440:85 , srv1159944459:85 , srv1170794044:85 , srv1184451494:85 , srv120088990:85 , srv124870115:85 , srv1273313777:85 , srv1287897741:85 , srv1327303540:85 , srv1339099818:85 , srv1359195055:85 , srv1411979577:85 , srv1559787211:85 , srv1617121609:85 , srv1650362002:85 , srv1663914481:85 , srv1670637447:85 , srv167562419:85 , srv1675758893:85 , srv1710491515:85 , srv1722550716:85 , srv1725901793:85 , srv1733118208:85 , srv1759926323:85 , srv1819025104:85 , srv1837756566:85 , srv1876507878:85 , srv1880036899:85 , srv1906073214:85 , srv1960774649:85 , srv2035309810:85 , srv205341357:85 , srv2054453124:85 , srv213397404:85 , srv22155102:85 , srv274291077:85 , srv291158136:85 , srv314072677:85 , srv335434584:85 , srv397261457:85 , srv423756036:85 , srv424302764:85 , srv459381803:85 , srv554761947:85 , srv639766334:85 , srv641665145:85 , srv662339871:85 , srv706178541:85 , srv710428285:85 , srv744611412:85 , srv966329852:85 , srv982240109:85 , srv985256531:85 , srv1228959325:86 , srv1229734258:86 , srv1495253164:86 , srv1895425474:87 , srv2102715508:621 } 2024-01-25 12:14:41,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 12:14:41,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv423756036=76, srv589737013=82, srv1273313777=16, srv115983992=8, srv1229734258=14, srv1827581363=46, srv1876507878=48, srv966329852=97, srv1339099818=20, srv1733118208=42, srv1517104167=27, srv1559787211=28, srv1159944459=9, srv397261457=74, srv1722550716=39, srv662339871=86, srv1027645286=1, srv167562419=34, srv201777660=54, srv785425627=93, srv354048751=71, srv1107580219=6, srv1303793754=18, srv205341357=59, srv124870115=15, srv1716775928=38, srv1758317488=43, srv310220447=68, srv641665145=84, srv1960774649=53, srv1906073214=52, srv740446547=91, srv2018184644=55, srv368379677=73, srv2054453124=60, srv1327303540=19, srv1013832330=0, srv1088956850=5, srv1170794044=10, srv1067442915=3, srv2035309810=57, srv459381803=78, srv1359195055=21, srv120088990=12, srv335434584=70, srv466877395=79, srv554761947=81, srv1617121609=30, srv22155102=63, srv1444009347=24, srv365240527=72, srv1228959325=13, srv1675758893=35, srv2038035955=58, srv1411979577=23, srv1759926323=44, srv268584938=64, srv649483775=85, srv1725901793=40, srv1895425474=51, srv985256531=99, srv910240561=96, srv702487129=88, srv982240109=98, srv416857571=75, srv856365713=94, srv424302764=77, srv1650362002=31, srv1880036899=49, srv1184451494=11, srv1685388018=36, srv1287897741=17, srv1663914481=32, srv1731631592=41, srv1670637447=33, srv1495253164=26, srv1405242608=22, srv2027905388=56, srv1819025104=45, srv314072677=69, srv274291077=65, srv1140385440=7, srv1889481870=50, srv639766334=83, srv907153600=95, srv706178541=89, srv291158136=67, srv1837756566=47, srv1571474523=29, srv2102715508=61, srv536285751=80, srv698759558=87, srv1710491515=37, srv213397404=62, srv275352161=66, srv1046375347=2, srv1453544184=25, srv710428285=90, srv744611412=92, srv1079958520=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 12:14:42,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 12:14:42,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 12:14:42,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 12:14:42,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 12:14:42,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-25 12:14:42,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-25 12:14:42,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-25 12:14:42,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-25 12:14:42,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-25 12:14:42,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-25 12:14:42,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-25 12:14:42,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-25 12:14:42,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-25 12:14:42,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-25 12:14:42,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-25 12:14:42,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-25 12:14:42,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-25 12:14:42,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-25 12:14:42,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-25 12:14:42,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-25 12:14:42,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-25 12:14:42,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-25 12:14:42,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-25 12:14:42,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-25 12:14:42,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-25 12:14:42,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-25 12:14:42,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-25 12:14:42,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-25 12:14:42,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-25 12:14:42,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-25 12:14:42,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-25 12:14:42,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-25 12:14:42,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-25 12:14:42,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-25 12:14:42,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-25 12:14:42,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-25 12:14:42,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-25 12:14:42,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-25 12:14:42,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-25 12:14:42,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-25 12:14:42,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-25 12:14:42,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-25 12:14:42,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-25 12:14:42,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-25 12:14:42,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-25 12:14:42,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-25 12:14:42,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-25 12:14:42,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-25 12:14:42,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-25 12:14:42,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-25 12:14:42,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-25 12:14:42,116 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:14:42,116 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-25 12:14:42,116 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.194023733266469E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-25 12:14:42,392 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 368 ms to try 5000 different iterations. Found a solution that moves 44 regions; Going from a computed imbalance of 4.194023733266469E-5 to a new imbalance of 5.9505875785642805E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0012535023884839248); MoveCostFunction : (multiplier=7.0, imbalance=0.004888888888888889); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:14:42,393 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1895425474:82 , srv1444009347:83 , srv1495253164:83 , srv1559787211:83 , srv1617121609:83 , srv205341357:83 , srv268584938:83 , srv639766334:83 , srv641665145:83 , srv744611412:83 , srv1013832330:84 , srv1067442915:84 , srv1079958520:84 , srv1088956850:84 , srv1107580219:84 , srv1140385440:84 , srv115983992:84 , srv1159944459:84 , srv1170794044:84 , srv1184451494:84 , srv120088990:84 , srv1228959325:84 , srv1229734258:84 , srv124870115:84 , srv1273313777:84 , srv1287897741:84 , srv1303793754:84 , srv1327303540:84 , srv1339099818:84 , srv1359195055:84 , srv1405242608:84 , srv1411979577:84 , srv1517104167:84 , srv1571474523:84 , srv1650362002:84 , srv1663914481:84 , srv167562419:84 , srv1675758893:84 , srv1685388018:84 , srv1710491515:84 , srv1716775928:84 , srv1722550716:84 , srv1725901793:84 , srv1733118208:84 , srv1758317488:84 , srv1759926323:84 , srv1819025104:84 , srv1827581363:84 , srv1837756566:84 , srv1876507878:84 , srv1880036899:84 , srv1889481870:84 , srv1906073214:84 , srv1960774649:84 , srv201777660:84 , srv2018184644:84 , srv2027905388:84 , srv2035309810:84 , srv2038035955:84 , srv2054453124:84 , srv213397404:84 , srv22155102:84 , srv274291077:84 , srv275352161:84 , srv291158136:84 , srv310220447:84 , srv314072677:84 , srv335434584:84 , srv354048751:84 , srv397261457:84 , srv416857571:84 , srv423756036:84 , srv424302764:84 , srv459381803:84 , srv466877395:84 , srv554761947:84 , srv589737013:84 , srv662339871:84 , srv698759558:84 , srv702487129:84 , srv706178541:84 , srv710428285:84 , srv740446547:84 , srv785425627:84 , srv907153600:84 , srv910240561:84 , srv982240109:84 , srv985256531:84 , srv1027645286:85 , srv1046375347:85 , srv1453544184:85 , srv1670637447:85 , srv1731631592:85 , srv365240527:85 , srv368379677:85 , srv536285751:85 , srv649483775:85 , srv856365713:85 , srv966329852:85 , srv2102715508:684 } 2024-01-25 12:14:42,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 12:14:42,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv423756036=76, srv589737013=82, srv1273313777=16, srv115983992=8, srv1229734258=14, srv1827581363=46, srv1876507878=48, srv966329852=97, srv1339099818=20, srv1733118208=42, srv1517104167=27, srv1559787211=28, srv1159944459=9, srv397261457=74, srv1722550716=39, srv662339871=86, srv1027645286=1, srv167562419=34, srv201777660=54, srv785425627=93, srv354048751=71, srv1107580219=6, srv1303793754=18, srv205341357=59, srv124870115=15, srv1716775928=38, srv1758317488=43, srv310220447=68, srv641665145=84, srv1960774649=53, srv1906073214=52, srv740446547=91, srv2018184644=55, srv368379677=73, srv2054453124=60, srv1327303540=19, srv1013832330=0, srv1088956850=5, srv1170794044=10, srv1067442915=3, srv2035309810=57, srv459381803=78, srv1359195055=21, srv120088990=12, srv335434584=70, srv466877395=79, srv554761947=81, srv1617121609=30, srv22155102=63, srv1444009347=24, srv365240527=72, srv1228959325=13, srv1675758893=35, srv2038035955=58, srv1411979577=23, srv1759926323=44, srv268584938=64, srv649483775=85, srv1725901793=40, srv1895425474=51, srv985256531=99, srv910240561=96, srv702487129=88, srv982240109=98, srv416857571=75, srv856365713=94, srv424302764=77, srv1650362002=31, srv1880036899=49, srv1184451494=11, srv1685388018=36, srv1287897741=17, srv1663914481=32, srv1731631592=41, srv1670637447=33, srv1495253164=26, srv1405242608=22, srv2027905388=56, srv1819025104=45, srv314072677=69, srv274291077=65, srv1140385440=7, srv1889481870=50, srv639766334=83, srv907153600=95, srv706178541=89, srv291158136=67, srv1837756566=47, srv1571474523=29, srv2102715508=61, srv536285751=80, srv698759558=87, srv1710491515=37, srv213397404=62, srv275352161=66, srv1046375347=2, srv1453544184=25, srv710428285=90, srv744611412=92, srv1079958520=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 12:14:42,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 12:14:42,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 12:14:42,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 12:14:42,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 12:14:42,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 12:14:42,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 12:14:42,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 12:14:42,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 12:14:42,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 12:14:42,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 12:14:42,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 12:14:42,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 12:14:42,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-25 12:14:42,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-25 12:14:42,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-25 12:14:42,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-25 12:14:42,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-25 12:14:42,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-25 12:14:42,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-25 12:14:42,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-25 12:14:42,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-25 12:14:42,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-25 12:14:42,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-25 12:14:42,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-25 12:14:42,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-25 12:14:42,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-25 12:14:42,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-25 12:14:42,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-25 12:14:42,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-25 12:14:42,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-25 12:14:42,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-25 12:14:42,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-25 12:14:42,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-25 12:14:42,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-25 12:14:42,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-25 12:14:42,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-25 12:14:42,569 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=5.642492993526674E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0012535023884839248); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:14:42,570 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1895425474:82 , srv1444009347:83 , srv1495253164:83 , srv1559787211:83 , srv1617121609:83 , srv205341357:83 , srv268584938:83 , srv639766334:83 , srv641665145:83 , srv744611412:83 , srv1013832330:84 , srv1067442915:84 , srv1079958520:84 , srv1088956850:84 , srv1107580219:84 , srv1140385440:84 , srv115983992:84 , srv1159944459:84 , srv1170794044:84 , srv1184451494:84 , srv120088990:84 , srv1228959325:84 , srv1229734258:84 , srv124870115:84 , srv1273313777:84 , srv1287897741:84 , srv1303793754:84 , srv1327303540:84 , srv1339099818:84 , srv1359195055:84 , srv1405242608:84 , srv1411979577:84 , srv1517104167:84 , srv1571474523:84 , srv1650362002:84 , srv1663914481:84 , srv167562419:84 , srv1675758893:84 , srv1685388018:84 , srv1710491515:84 , srv1716775928:84 , srv1722550716:84 , srv1725901793:84 , srv1733118208:84 , srv1758317488:84 , srv1759926323:84 , srv1819025104:84 , srv1827581363:84 , srv1837756566:84 , srv1876507878:84 , srv1880036899:84 , srv1889481870:84 , srv1906073214:84 , srv1960774649:84 , srv201777660:84 , srv2018184644:84 , srv2027905388:84 , srv2035309810:84 , srv2038035955:84 , srv2054453124:84 , srv213397404:84 , srv22155102:84 , srv274291077:84 , srv275352161:84 , srv291158136:84 , srv310220447:84 , srv314072677:84 , srv335434584:84 , srv354048751:84 , srv397261457:84 , srv416857571:84 , srv423756036:84 , srv424302764:84 , srv459381803:84 , srv466877395:84 , srv554761947:84 , srv589737013:84 , srv662339871:84 , srv698759558:84 , srv702487129:84 , srv706178541:84 , srv710428285:84 , srv740446547:84 , srv785425627:84 , srv907153600:84 , srv910240561:84 , srv982240109:84 , srv985256531:84 , srv1027645286:85 , srv1046375347:85 , srv1453544184:85 , srv1670637447:85 , srv1731631592:85 , srv365240527:85 , srv368379677:85 , srv536285751:85 , srv649483775:85 , srv856365713:85 , srv966329852:85 , srv2102715508:684 } 2024-01-25 12:14:42,602 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=275 (was 275), ProcessCount=170 (was 170), AvailableMemoryMB=9074 (was 9995) 2024-01-25 12:14:42,613 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=275, ProcessCount=170, AvailableMemoryMB=9072 2024-01-25 12:14:42,614 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-25 12:14:42,614 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-01-25 12:14:42,622 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1058496510:84 , srv1081997765:84 , srv1123684646:84 , srv11285684:84 , srv1183774250:84 , srv1203073556:84 , srv1245203301:84 , srv1289969007:84 , srv1479791934:84 , srv1515919057:84 , srv1563451065:84 , srv1612918678:84 , srv180490426:84 , srv1889178486:84 , srv1931194854:84 , srv1963496678:84 , srv243496248:84 , srv355693886:84 , srv391602:84 , srv431727575:84 , srv460107815:84 , srv522063542:84 , srv721100819:84 , srv727396098:84 , srv765716811:84 , srv849485572:84 , srv895689292:84 , srv939106593:84 , srv986552408:84 , srv1425645288:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-01-25 12:14:42,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 12:14:42,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1563451065=11, srv1963496678=16, srv849485572=26, srv1203073556=5, srv986552408=29, srv727396098=24, srv460107815=21, srv522063542=22, srv1889178486=14, srv431727575=20, srv1081997765=1, srv765716811=25, srv11285684=3, srv1515919057=10, srv391602=19, srv180490426=13, srv1058496510=0, srv1183774250=4, srv1479791934=9, srv355693886=18, srv1123684646=2, srv895689292=27, srv1425645288=8, srv243496248=17, srv1612918678=12, srv939106593=28, srv721100819=23, srv1931194854=15, srv1245203301=6, srv1289969007=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 12:14:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 12:14:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 12:14:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 12:14:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 12:14:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 12:14:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 12:14:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 12:14:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 12:14:42,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 12:14:42,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-25 12:14:42,653 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.2312229682687146, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:14:42,653 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912921619287168, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.2312229682687146, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-01-25 12:16:42,638 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6645099 different iterations. Found a solution that moves 1854 regions; Going from a computed imbalance of 0.9912921619287168 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:16:42,645 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1058496510:90 , srv1081997765:90 , srv1123684646:90 , srv11285684:90 , srv1183774250:90 , srv1203073556:90 , srv1245203301:90 , srv1289969007:90 , srv1425645288:90 , srv1479791934:90 , srv1515919057:90 , srv1563451065:90 , srv1612918678:90 , srv180490426:90 , srv1889178486:90 , srv1931194854:90 , srv1963496678:90 , srv243496248:90 , srv355693886:90 , srv391602:90 , srv431727575:90 , srv460107815:90 , srv522063542:90 , srv721100819:90 , srv727396098:90 , srv765716811:90 , srv849485572:90 , srv895689292:90 , srv939106593:90 , srv986552408:90 } 2024-01-25 12:16:42,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 12:16:42,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1563451065=11, srv1963496678=16, srv849485572=26, srv1203073556=5, srv986552408=29, srv727396098=24, srv460107815=21, srv522063542=22, srv1889178486=14, srv431727575=20, srv1081997765=1, srv765716811=25, srv11285684=3, srv1515919057=10, srv391602=19, srv180490426=13, srv1058496510=0, srv1183774250=4, srv1479791934=9, srv355693886=18, srv1123684646=2, srv895689292=27, srv1425645288=8, srv243496248=17, srv1612918678=12, srv939106593=28, srv721100819=23, srv1931194854=15, srv1245203301=6, srv1289969007=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 12:16:42,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 12:16:42,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 12:16:42,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 12:16:42,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 12:16:42,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 12:16:42,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 12:16:42,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 12:16:42,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 12:16:42,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 12:16:42,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 12:16:42,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 12:16:42,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 12:16:42,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 12:16:42,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 12:16:42,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 12:16:42,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-25 12:16:42,666 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 12:16:42,666 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1058496510:90 , srv1081997765:90 , srv1123684646:90 , srv11285684:90 , srv1183774250:90 , srv1203073556:90 , srv1245203301:90 , srv1289969007:90 , srv1425645288:90 , srv1479791934:90 , srv1515919057:90 , srv1563451065:90 , srv1612918678:90 , srv180490426:90 , srv1889178486:90 , srv1931194854:90 , srv1963496678:90 , srv243496248:90 , srv355693886:90 , srv391602:90 , srv431727575:90 , srv460107815:90 , srv522063542:90 , srv721100819:90 , srv727396098:90 , srv765716811:90 , srv849485572:90 , srv895689292:90 , srv939106593:90 , srv986552408:90 } 2024-01-25 12:16:42,679 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=250 (was 275), ProcessCount=169 (was 170), AvailableMemoryMB=8436 (was 9072)