2023-05-19 20:53:26,834 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-19 20:53:27,149 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-19 20:53:27,263 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-19 20:53:27,327 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-19 20:53:27,337 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-19 20:53:27,355 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=391, ProcessCount=172, AvailableMemoryMB=8657 2023-05-19 20:53:27,357 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-19 20:53:27,357 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-19 20:53:27,498 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1021095887:84 , srv1055371113:84 , srv1061220919:84 , srv1065568386:84 , srv1074435512:84 , srv1075199380:84 , srv1076033619:84 , srv1079822256:84 , srv1091388866:84 , srv1120485660:84 , srv1124417543:84 , srv1125463396:84 , srv1133841743:84 , srv1186109526:84 , srv1220183021:84 , srv1255424961:84 , srv1257942641:84 , srv1339444637:84 , srv134982537:84 , srv1361562873:84 , srv1388803885:84 , srv1389159407:84 , srv1399393919:84 , srv1459638471:84 , srv1473105983:84 , srv1543635117:84 , srv1544977718:84 , srv1559423834:84 , srv156279995:84 , srv1596708188:84 , srv1609110706:84 , srv1661528166:84 , srv1701860532:84 , srv1722814966:84 , srv1723186684:84 , srv1749701756:84 , srv1764481515:84 , srv1795319853:84 , srv1813841314:84 , srv1823245342:84 , srv1828536048:84 , srv1829759178:84 , srv1856726378:84 , srv1858934898:84 , srv1899623830:84 , srv1900450237:84 , srv1912034675:84 , srv1923383578:84 , srv193552618:84 , srv1951285527:84 , srv1958662034:84 , srv1961725001:84 , srv1965876698:84 , srv20025411:84 , srv2023116504:84 , srv2040769922:84 , srv2073830444:84 , srv2081523747:84 , srv2106068747:84 , srv2117366843:84 , srv2138767601:84 , srv2142412766:84 , srv225865616:84 , srv263883458:84 , srv280558397:84 , srv286729869:84 , srv292349616:84 , srv315809721:84 , srv319276580:84 , srv319734629:84 , srv330471413:84 , srv354061417:84 , srv358867555:84 , srv36481867:84 , srv42293653:84 , srv43062492:84 , srv492586718:84 , srv497211644:84 , srv597718478:84 , srv629608440:84 , srv653500979:84 , srv686932666:84 , srv700132935:84 , srv704672012:84 , srv752802284:84 , srv759461183:84 , srv771175222:84 , srv784673670:84 , srv841853828:84 , srv875956236:84 , srv884117685:84 , srv899609166:84 , srv907942435:84 , srv919307461:84 , srv923688601:84 , srv933123168:84 , srv935608429:84 , srv964992621:84 , srv986013611:84 , srv1149741582:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-19 20:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-19 20:53:27,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv907942435=93, srv923688601=95, srv2073830444=57, srv899609166=92, srv1559423834=28, srv1795319853=38, srv1389159407=22, srv1133841743=12, srv841853828=89, srv1596708188=30, srv1813841314=39, srv1124417543=10, srv752802284=85, srv315809721=68, srv935608429=97, srv933123168=96, srv263883458=64, srv1120485660=9, srv1055371113=1, srv700132935=83, srv1923383578=48, srv1823245342=40, srv1764481515=37, srv704672012=84, srv2142412766=62, srv1074435512=4, srv330471413=71, srv1075199380=5, srv1076033619=6, srv629608440=80, srv686932666=82, srv20025411=54, srv1399393919=23, srv1459638471=24, srv319734629=70, srv2106068747=59, srv492586718=77, srv771175222=87, srv2138767601=61, srv1722814966=34, srv1186109526=14, srv875956236=90, srv1899623830=45, srv286729869=66, srv354061417=72, srv1749701756=36, srv964992621=98, srv1958662034=51, srv225865616=63, srv1544977718=27, srv759461183=86, srv1473105983=25, srv1900450237=46, srv1723186684=35, srv1388803885=21, srv653500979=81, srv36481867=74, srv43062492=76, srv597718478=79, srv1828536048=41, srv2081523747=58, srv1125463396=11, srv292349616=67, srv919307461=94, srv1091388866=8, srv784673670=88, srv986013611=99, srv1220183021=15, srv1912034675=47, srv1361562873=20, srv193552618=49, srv280558397=65, srv1856726378=43, srv1021095887=0, srv1829759178=42, srv1965876698=53, srv2117366843=60, srv884117685=91, srv1079822256=7, srv42293653=75, srv497211644=78, srv358867555=73, srv1339444637=18, srv156279995=29, srv2023116504=55, srv1257942641=17, srv1701860532=33, srv1149741582=13, srv1609110706=31, srv1661528166=32, srv2040769922=56, srv1543635117=26, srv1065568386=3, srv1255424961=16, srv1961725001=52, srv134982537=19, srv319276580=69, srv1858934898=44, srv1951285527=50, srv1061220919=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-19 20:53:27,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-19 20:53:27,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-19 20:53:27,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-19 20:53:27,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-19 20:53:27,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-19 20:53:27,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-19 20:53:27,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-19 20:53:27,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-19 20:53:27,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-19 20:53:27,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-19 20:53:27,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-19 20:53:27,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-19 20:53:27,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-19 20:53:27,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-19 20:53:27,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-19 20:53:27,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-19 20:53:27,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-19 20:53:27,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-19 20:53:27,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-19 20:53:27,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-19 20:53:27,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-19 20:53:27,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-19 20:53:27,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-19 20:53:27,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-19 20:53:27,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-19 20:53:27,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-19 20:53:27,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-19 20:53:27,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-19 20:53:27,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-19 20:53:27,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-19 20:53:27,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-19 20:53:27,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-19 20:53:27,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-19 20:53:27,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-19 20:53:27,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-19 20:53:27,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-19 20:53:27,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-19 20:53:27,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-19 20:53:27,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-19 20:53:27,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-19 20:53:27,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-19 20:53:27,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-19 20:53:27,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-19 20:53:27,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-19 20:53:27,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-19 20:53:27,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-19 20:53:27,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-19 20:53:27,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-19 20:53:27,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-19 20:53:27,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-19 20:53:27,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-19 20:53:27,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-19 20:53:27,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-19 20:53:27,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-19 20:53:27,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-19 20:53:27,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-19 20:53:27,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-19 20:53:27,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-19 20:53:27,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-19 20:53:27,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-19 20:53:27,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-19 20:53:27,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-19 20:53:27,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-19 20:53:27,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-19 20:53:27,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-19 20:53:27,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-19 20:53:27,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-19 20:53:27,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-19 20:53:27,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-19 20:53:27,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-19 20:53:27,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-19 20:53:27,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-19 20:53:27,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-19 20:53:27,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-19 20:53:27,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-19 20:53:27,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-19 20:53:27,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-19 20:53:27,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-19 20:53:27,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-19 20:53:27,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-19 20:53:27,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-19 20:53:27,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-19 20:53:27,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-19 20:53:27,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-19 20:53:27,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-19 20:53:27,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-19 20:53:27,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-19 20:53:27,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-19 20:53:27,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-19 20:53:27,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-19 20:53:27,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-19 20:53:27,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-19 20:53:27,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-19 20:53:27,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-19 20:53:27,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-19 20:53:27,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-19 20:53:27,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-19 20:53:27,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-19 20:53:27,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-19 20:53:27,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-19 20:53:27,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-19 20:53:27,770 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-19 20:53:27,770 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-19 20:53:27,770 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-19 20:53:28,120 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 497 ms to try 5000 different iterations. Found a solution that moves 4717 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17734049016426717. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040819024879567095, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05755200644258007, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5241111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040819024879567095, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-19 20:53:28,135 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv923688601:81 , srv1079822256:82 , srv1124417543:82 , srv1389159407:82 , srv1961725001:82 , srv330471413:82 , srv156279995:83 , srv1823245342:83 , srv771175222:83 , srv907942435:83 , srv919307461:83 , srv1609110706:84 , srv1923383578:84 , srv1958662034:84 , srv2117366843:84 , srv43062492:84 , srv784673670:84 , srv986013611:84 , srv1021095887:85 , srv1065568386:85 , srv1255424961:85 , srv1749701756:85 , srv1764481515:85 , srv1829759178:85 , srv1900450237:85 , srv193552618:85 , srv1951285527:85 , srv2106068747:85 , srv2142412766:85 , srv315809721:85 , srv319276580:85 , srv354061417:85 , srv358867555:85 , srv492586718:85 , srv759461183:85 , srv1061220919:86 , srv1075199380:86 , srv1091388866:86 , srv1120485660:86 , srv1459638471:86 , srv1544977718:86 , srv1701860532:86 , srv1722814966:86 , srv1795319853:86 , srv1858934898:86 , srv20025411:86 , srv292349616:86 , srv319734629:86 , srv629608440:86 , srv686932666:86 , srv700132935:86 , srv933123168:86 , srv1220183021:87 , srv1388803885:87 , srv1596708188:87 , srv1723186684:87 , srv1828536048:87 , srv1856726378:87 , srv1912034675:87 , srv1965876698:87 , srv2023116504:87 , srv2040769922:87 , srv2073830444:87 , srv2081523747:87 , srv286729869:87 , srv42293653:87 , srv497211644:87 , srv597718478:87 , srv841853828:87 , srv884117685:87 , srv899609166:87 , srv1055371113:88 , srv1076033619:88 , srv1186109526:88 , srv1361562873:88 , srv1473105983:88 , srv1559423834:88 , srv2138767601:88 , srv225865616:88 , srv263883458:88 , srv280558397:88 , srv653500979:88 , srv704672012:88 , srv752802284:88 , srv1133841743:89 , srv1339444637:89 , srv134982537:89 , srv1661528166:89 , srv36481867:89 , srv875956236:89 , srv964992621:89 , srv1074435512:90 , srv1257942641:90 , srv1399393919:90 , srv1813841314:90 , srv935608429:90 , srv1125463396:91 , srv1899623830:91 , srv1543635117:94 , srv1149741582:453 } 2023-05-19 20:53:28,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-19 20:53:28,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv907942435=93, srv923688601=95, srv2073830444=57, srv899609166=92, srv1559423834=28, srv1795319853=38, srv1389159407=22, srv1133841743=12, srv841853828=89, srv1596708188=30, srv1813841314=39, srv1124417543=10, srv752802284=85, srv315809721=68, srv935608429=97, srv933123168=96, srv263883458=64, srv1120485660=9, srv1055371113=1, srv700132935=83, srv1923383578=48, srv1823245342=40, srv1764481515=37, srv704672012=84, srv2142412766=62, srv1074435512=4, srv330471413=71, srv1075199380=5, srv1076033619=6, srv629608440=80, srv686932666=82, srv20025411=54, srv1399393919=23, srv1459638471=24, srv319734629=70, srv2106068747=59, srv492586718=77, srv771175222=87, srv2138767601=61, srv1722814966=34, srv1186109526=14, srv875956236=90, srv1899623830=45, srv286729869=66, srv354061417=72, srv1749701756=36, srv964992621=98, srv1958662034=51, srv225865616=63, srv1544977718=27, srv759461183=86, srv1473105983=25, srv1900450237=46, srv1723186684=35, srv1388803885=21, srv653500979=81, srv36481867=74, srv43062492=76, srv597718478=79, srv1828536048=41, srv2081523747=58, srv1125463396=11, srv292349616=67, srv919307461=94, srv1091388866=8, srv784673670=88, srv986013611=99, srv1220183021=15, srv1912034675=47, srv1361562873=20, srv193552618=49, srv280558397=65, srv1856726378=43, srv1021095887=0, srv1829759178=42, srv1965876698=53, srv2117366843=60, srv884117685=91, srv1079822256=7, srv42293653=75, srv497211644=78, srv358867555=73, srv1339444637=18, srv156279995=29, srv2023116504=55, srv1257942641=17, srv1701860532=33, srv1149741582=13, srv1609110706=31, srv1661528166=32, srv2040769922=56, srv1543635117=26, srv1065568386=3, srv1255424961=16, srv1961725001=52, srv134982537=19, srv319276580=69, srv1858934898=44, srv1951285527=50, srv1061220919=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-19 20:53:28,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-19 20:53:28,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-19 20:53:28,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-19 20:53:28,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-19 20:53:28,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-19 20:53:28,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-19 20:53:28,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-19 20:53:28,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-19 20:53:28,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-19 20:53:28,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-19 20:53:28,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-19 20:53:28,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-19 20:53:28,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-19 20:53:28,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-19 20:53:28,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-19 20:53:28,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-19 20:53:28,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-19 20:53:28,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-19 20:53:28,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-19 20:53:28,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-19 20:53:28,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-19 20:53:28,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-19 20:53:28,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-19 20:53:28,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-19 20:53:28,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-19 20:53:28,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-19 20:53:28,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-19 20:53:28,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-19 20:53:28,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-19 20:53:28,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-19 20:53:28,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-19 20:53:28,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-19 20:53:28,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-19 20:53:28,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-19 20:53:28,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-19 20:53:28,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-19 20:53:28,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-19 20:53:28,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-19 20:53:28,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-19 20:53:28,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-19 20:53:28,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-19 20:53:28,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-19 20:53:28,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-19 20:53:28,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-19 20:53:28,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-19 20:53:28,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-19 20:53:28,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-19 20:53:28,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-19 20:53:28,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-19 20:53:28,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-19 20:53:28,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-19 20:53:28,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-19 20:53:28,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-19 20:53:28,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-19 20:53:28,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-19 20:53:28,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-19 20:53:28,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-19 20:53:28,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-19 20:53:28,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-19 20:53:28,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-19 20:53:28,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-19 20:53:28,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-19 20:53:28,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-19 20:53:28,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-19 20:53:28,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-19 20:53:28,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-19 20:53:28,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-19 20:53:28,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-19 20:53:28,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-19 20:53:28,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-19 20:53:28,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-19 20:53:28,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-19 20:53:28,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-19 20:53:28,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-19 20:53:28,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-19 20:53:28,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-19 20:53:28,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-19 20:53:28,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-19 20:53:28,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-19 20:53:28,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-19 20:53:28,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-19 20:53:28,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-19 20:53:28,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-19 20:53:28,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-19 20:53:28,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-19 20:53:28,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-19 20:53:28,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-19 20:53:28,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-19 20:53:28,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-19 20:53:28,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-19 20:53:28,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-19 20:53:28,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-19 20:53:28,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-19 20:53:28,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-19 20:53:28,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-19 20:53:28,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-19 20:53:28,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-19 20:53:28,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-19 20:53:28,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-19 20:53:28,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-19 20:53:28,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-19 20:53:28,351 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040819024879567095, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05755200644258007, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040819024879567095, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-19 20:53:28,351 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-19 20:53:28,351 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1773074610243212, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040819024879567095, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05755200644258007, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040819024879567095, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17266666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-19 20:53:28,681 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 481 ms to try 5000 different iterations. Found a solution that moves 2608 regions; Going from a computed imbalance of 0.1773074610243212 to a new imbalance of 0.015896495043057193. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02093106518936248); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03668349782048706, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2897777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02093106518936248); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045083333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-19 20:53:28,694 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1899623830:80 , srv1125463396:81 , srv1257942641:81 , srv1339444637:81 , srv1543635117:81 , srv935608429:81 , srv1074435512:82 , srv1133841743:82 , srv134982537:82 , srv36481867:82 , srv1055371113:83 , srv1388803885:83 , srv1399393919:83 , srv2023116504:83 , srv2040769922:83 , srv2138767601:83 , srv964992621:83 , srv1186109526:84 , srv1220183021:84 , srv1361562873:84 , srv1544977718:84 , srv1661528166:84 , srv1723186684:84 , srv1813841314:84 , srv1856726378:84 , srv1912034675:84 , srv225865616:84 , srv263883458:84 , srv280558397:84 , srv292349616:84 , srv42293653:84 , srv686932666:84 , srv704672012:84 , srv752802284:84 , srv875956236:84 , srv899609166:84 , srv1065568386:85 , srv1722814966:85 , srv1829759178:85 , srv1900450237:85 , srv2073830444:85 , srv497211644:85 , srv597718478:85 , srv653500979:85 , srv1021095887:86 , srv1076033619:86 , srv1459638471:86 , srv1473105983:86 , srv1559423834:86 , srv1596708188:86 , srv1701860532:86 , srv1749701756:86 , srv1795319853:86 , srv1858934898:86 , srv193552618:86 , srv1965876698:86 , srv20025411:86 , srv2081523747:86 , srv2142412766:86 , srv629608440:86 , srv933123168:86 , srv1061220919:87 , srv1120485660:87 , srv1828536048:87 , srv1958662034:87 , srv286729869:87 , srv319276580:87 , srv319734629:87 , srv354061417:87 , srv700132935:87 , srv759461183:87 , srv1075199380:88 , srv1255424961:88 , srv1609110706:88 , srv315809721:88 , srv43062492:88 , srv492586718:88 , srv771175222:88 , srv784673670:88 , srv841853828:88 , srv884117685:88 , srv986013611:88 , srv1079822256:89 , srv1091388866:89 , srv1389159407:89 , srv1764481515:89 , srv1923383578:89 , srv1961725001:89 , srv2106068747:89 , srv2117366843:89 , srv358867555:89 , srv907942435:89 , srv919307461:89 , srv156279995:90 , srv1823245342:90 , srv330471413:90 , srv923688601:90 , srv1124417543:92 , srv1951285527:92 , srv1149741582:507 } 2023-05-19 20:53:28,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-19 20:53:28,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv907942435=93, srv923688601=95, srv2073830444=57, srv899609166=92, srv1559423834=28, srv1795319853=38, srv1389159407=22, srv1133841743=12, srv841853828=89, srv1596708188=30, srv1813841314=39, srv1124417543=10, srv752802284=85, srv315809721=68, srv935608429=97, srv933123168=96, srv263883458=64, srv1120485660=9, srv1055371113=1, srv700132935=83, srv1923383578=48, srv1823245342=40, srv1764481515=37, srv704672012=84, srv2142412766=62, srv1074435512=4, srv330471413=71, srv1075199380=5, srv1076033619=6, srv629608440=80, srv686932666=82, srv20025411=54, srv1399393919=23, srv1459638471=24, srv319734629=70, srv2106068747=59, srv492586718=77, srv771175222=87, srv2138767601=61, srv1722814966=34, srv1186109526=14, srv875956236=90, srv1899623830=45, srv286729869=66, srv354061417=72, srv1749701756=36, srv964992621=98, srv1958662034=51, srv225865616=63, srv1544977718=27, srv759461183=86, srv1473105983=25, srv1900450237=46, srv1723186684=35, srv1388803885=21, srv653500979=81, srv36481867=74, srv43062492=76, srv597718478=79, srv1828536048=41, srv2081523747=58, srv1125463396=11, srv292349616=67, srv919307461=94, srv1091388866=8, srv784673670=88, srv986013611=99, srv1220183021=15, srv1912034675=47, srv1361562873=20, srv193552618=49, srv280558397=65, srv1856726378=43, srv1021095887=0, srv1829759178=42, srv1965876698=53, srv2117366843=60, srv884117685=91, srv1079822256=7, srv42293653=75, srv497211644=78, srv358867555=73, srv1339444637=18, srv156279995=29, srv2023116504=55, srv1257942641=17, srv1701860532=33, srv1149741582=13, srv1609110706=31, srv1661528166=32, srv2040769922=56, srv1543635117=26, srv1065568386=3, srv1255424961=16, srv1961725001=52, srv134982537=19, srv319276580=69, srv1858934898=44, srv1951285527=50, srv1061220919=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-19 20:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-19 20:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-19 20:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-19 20:53:28,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-19 20:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-19 20:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-19 20:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-19 20:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-19 20:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-19 20:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-19 20:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-19 20:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-19 20:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-19 20:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-19 20:53:28,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-19 20:53:28,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-19 20:53:28,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-19 20:53:28,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-19 20:53:28,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-19 20:53:28,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-19 20:53:28,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-19 20:53:28,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-19 20:53:28,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-19 20:53:28,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-19 20:53:28,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-19 20:53:28,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-19 20:53:28,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-19 20:53:28,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-19 20:53:28,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-19 20:53:28,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-19 20:53:28,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-19 20:53:28,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-19 20:53:28,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-19 20:53:28,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-19 20:53:28,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-19 20:53:28,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-19 20:53:28,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-19 20:53:28,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-19 20:53:28,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-19 20:53:28,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-19 20:53:28,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-19 20:53:28,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-19 20:53:28,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-19 20:53:28,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-19 20:53:28,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-19 20:53:28,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-19 20:53:28,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-19 20:53:28,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-19 20:53:28,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-19 20:53:28,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-19 20:53:28,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-19 20:53:28,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-19 20:53:28,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-19 20:53:28,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-19 20:53:28,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-19 20:53:28,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-19 20:53:28,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-19 20:53:28,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-19 20:53:28,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-19 20:53:28,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-19 20:53:28,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-19 20:53:28,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-19 20:53:28,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-19 20:53:28,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-19 20:53:28,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-19 20:53:28,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-19 20:53:28,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-19 20:53:28,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-19 20:53:28,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-19 20:53:28,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-19 20:53:28,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-19 20:53:28,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-19 20:53:28,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-19 20:53:28,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-19 20:53:28,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-19 20:53:28,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-19 20:53:28,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-19 20:53:28,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-19 20:53:28,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-19 20:53:28,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-19 20:53:28,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-19 20:53:28,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-19 20:53:28,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-19 20:53:28,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-19 20:53:28,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-19 20:53:28,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-19 20:53:28,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-19 20:53:28,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-19 20:53:28,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-19 20:53:28,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-19 20:53:28,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-19 20:53:28,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-19 20:53:28,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-19 20:53:28,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-19 20:53:28,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-19 20:53:28,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-19 20:53:28,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-19 20:53:28,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-19 20:53:28,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-19 20:53:28,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-19 20:53:28,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-19 20:53:28,910 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02093106518936248); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03668349782048706, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02093106518936248); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045083333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-19 20:53:28,910 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-19 20:53:28,910 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015878233436744053, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02093106518936248); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03668349782048706, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02093106518936248); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045083333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-19 20:53:29,237 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 473 ms to try 5000 different iterations. Found a solution that moves 960 regions; Going from a computed imbalance of 0.015878233436744053 to a new imbalance of 5.879176927281818E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0059824743002341195); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009380352942419313); MoveCostFunction : (multiplier=7.0, imbalance=0.10666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0059824743002341195); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-19 20:53:29,240 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1951285527:81 , srv841853828:82 , srv884117685:82 , srv1075199380:83 , srv1076033619:83 , srv1091388866:83 , srv1543635117:83 , srv1764481515:83 , srv1813841314:83 , srv286729869:83 , srv358867555:83 , srv1061220919:84 , srv1124417543:84 , srv1473105983:84 , srv1559423834:84 , srv1596708188:84 , srv1661528166:84 , srv1823245342:84 , srv1828536048:84 , srv2106068747:84 , srv492586718:84 , srv1120485660:85 , srv1125463396:85 , srv1186109526:85 , srv1255424961:85 , srv1361562873:85 , srv1399393919:85 , srv1459638471:85 , srv156279995:85 , srv1609110706:85 , srv1701860532:85 , srv1795319853:85 , srv1858934898:85 , srv1923383578:85 , srv1965876698:85 , srv2073830444:85 , srv2081523747:85 , srv2117366843:85 , srv225865616:85 , srv315809721:85 , srv319734629:85 , srv330471413:85 , srv597718478:85 , srv629608440:85 , srv653500979:85 , srv700132935:85 , srv752802284:85 , srv784673670:85 , srv875956236:85 , srv919307461:85 , srv933123168:85 , srv964992621:85 , srv986013611:85 , srv1021095887:86 , srv1055371113:86 , srv1074435512:86 , srv1079822256:86 , srv1257942641:86 , srv134982537:86 , srv1389159407:86 , srv1723186684:86 , srv1749701756:86 , srv1912034675:86 , srv193552618:86 , srv1958662034:86 , srv20025411:86 , srv2138767601:86 , srv2142412766:86 , srv263883458:86 , srv280558397:86 , srv319276580:86 , srv354061417:86 , srv42293653:86 , srv43062492:86 , srv497211644:86 , srv704672012:86 , srv759461183:86 , srv771175222:86 , srv899609166:86 , srv907942435:86 , srv935608429:86 , srv1065568386:87 , srv1133841743:87 , srv1220183021:87 , srv1388803885:87 , srv1544977718:87 , srv1722814966:87 , srv1856726378:87 , srv1899623830:87 , srv1900450237:87 , srv2023116504:87 , srv292349616:87 , srv686932666:87 , srv923688601:87 , srv1339444637:88 , srv1829759178:88 , srv1961725001:88 , srv2040769922:88 , srv36481867:89 , srv1149741582:551 } 2023-05-19 20:53:29,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-19 20:53:29,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv907942435=93, srv923688601=95, srv2073830444=57, srv899609166=92, srv1559423834=28, srv1795319853=38, srv1389159407=22, srv1133841743=12, srv841853828=89, srv1596708188=30, srv1813841314=39, srv1124417543=10, srv752802284=85, srv315809721=68, srv935608429=97, srv933123168=96, srv263883458=64, srv1120485660=9, srv1055371113=1, srv700132935=83, srv1923383578=48, srv1823245342=40, srv1764481515=37, srv704672012=84, srv2142412766=62, srv1074435512=4, srv330471413=71, srv1075199380=5, srv1076033619=6, srv629608440=80, srv686932666=82, srv20025411=54, srv1399393919=23, srv1459638471=24, srv319734629=70, srv2106068747=59, srv492586718=77, srv771175222=87, srv2138767601=61, srv1722814966=34, srv1186109526=14, srv875956236=90, srv1899623830=45, srv286729869=66, srv354061417=72, srv1749701756=36, srv964992621=98, srv1958662034=51, srv225865616=63, srv1544977718=27, srv759461183=86, srv1473105983=25, srv1900450237=46, srv1723186684=35, srv1388803885=21, srv653500979=81, srv36481867=74, srv43062492=76, srv597718478=79, srv1828536048=41, srv2081523747=58, srv1125463396=11, srv292349616=67, srv919307461=94, srv1091388866=8, srv784673670=88, srv986013611=99, srv1220183021=15, srv1912034675=47, srv1361562873=20, srv193552618=49, srv280558397=65, srv1856726378=43, srv1021095887=0, srv1829759178=42, srv1965876698=53, srv2117366843=60, srv884117685=91, srv1079822256=7, srv42293653=75, srv497211644=78, srv358867555=73, srv1339444637=18, srv156279995=29, srv2023116504=55, srv1257942641=17, srv1701860532=33, srv1149741582=13, srv1609110706=31, srv1661528166=32, srv2040769922=56, srv1543635117=26, srv1065568386=3, srv1255424961=16, srv1961725001=52, srv134982537=19, srv319276580=69, srv1858934898=44, srv1951285527=50, srv1061220919=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-19 20:53:29,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-19 20:53:29,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-19 20:53:29,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-19 20:53:29,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-19 20:53:29,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-19 20:53:29,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-19 20:53:29,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-19 20:53:29,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-19 20:53:29,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-19 20:53:29,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-19 20:53:29,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-19 20:53:29,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-19 20:53:29,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-19 20:53:29,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-19 20:53:29,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-19 20:53:29,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-19 20:53:29,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-19 20:53:29,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-19 20:53:29,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-19 20:53:29,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-19 20:53:29,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-19 20:53:29,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-19 20:53:29,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-19 20:53:29,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-19 20:53:29,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-19 20:53:29,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-19 20:53:29,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-19 20:53:29,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-19 20:53:29,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-19 20:53:29,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-19 20:53:29,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-19 20:53:29,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-19 20:53:29,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-19 20:53:29,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-19 20:53:29,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-19 20:53:29,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-19 20:53:29,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-19 20:53:29,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-19 20:53:29,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-19 20:53:29,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-19 20:53:29,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-19 20:53:29,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-19 20:53:29,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-19 20:53:29,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-19 20:53:29,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-19 20:53:29,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-19 20:53:29,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-19 20:53:29,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-19 20:53:29,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-19 20:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-19 20:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-19 20:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-19 20:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-19 20:53:29,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-19 20:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-19 20:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-19 20:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-19 20:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-19 20:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-19 20:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-19 20:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-19 20:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-19 20:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-19 20:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-19 20:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-19 20:53:29,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-19 20:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-19 20:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-19 20:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-19 20:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-19 20:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-19 20:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-19 20:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-19 20:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-19 20:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-19 20:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-19 20:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-19 20:53:29,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-19 20:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-19 20:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-19 20:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-19 20:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-19 20:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-19 20:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-19 20:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-19 20:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-19 20:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-19 20:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-19 20:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-19 20:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-19 20:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-19 20:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-19 20:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-19 20:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-19 20:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-19 20:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-19 20:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-19 20:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-19 20:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-19 20:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-19 20:53:29,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-19 20:53:29,425 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0059824743002341195); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009380352942419313); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0059824743002341195); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-19 20:53:29,425 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-19 20:53:29,426 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.81195629054634E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0059824743002341195); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009380352942419313); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0059824743002341195); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-19 20:53:29,836 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 528 ms to try 5000 different iterations. Found a solution that moves 257 regions; Going from a computed imbalance of 5.81195629054634E-4 to a new imbalance of 2.1456368029639813E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.028555555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-19 20:53:29,838 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1899623830:83 , srv36481867:83 , srv1074435512:84 , srv1075199380:84 , srv1091388866:84 , srv1120485660:84 , srv1124417543:84 , srv1133841743:84 , srv1255424961:84 , srv1339444637:84 , srv134982537:84 , srv1388803885:84 , srv1399393919:84 , srv1473105983:84 , srv1543635117:84 , srv1544977718:84 , srv1559423834:84 , srv156279995:84 , srv1701860532:84 , srv1828536048:84 , srv1923383578:84 , srv1951285527:84 , srv1961725001:84 , srv1965876698:84 , srv20025411:84 , srv2040769922:84 , srv2081523747:84 , srv2106068747:84 , srv2117366843:84 , srv263883458:84 , srv280558397:84 , srv315809721:84 , srv319276580:84 , srv319734629:84 , srv354061417:84 , srv497211644:84 , srv653500979:84 , srv700132935:84 , srv704672012:84 , srv752802284:84 , srv759461183:84 , srv875956236:84 , srv907942435:84 , srv923688601:84 , srv1021095887:85 , srv1055371113:85 , srv1061220919:85 , srv1065568386:85 , srv1079822256:85 , srv1125463396:85 , srv1186109526:85 , srv1220183021:85 , srv1257942641:85 , srv1361562873:85 , srv1389159407:85 , srv1459638471:85 , srv1596708188:85 , srv1609110706:85 , srv1661528166:85 , srv1722814966:85 , srv1723186684:85 , srv1749701756:85 , srv1764481515:85 , srv1795319853:85 , srv1813841314:85 , srv1823245342:85 , srv1829759178:85 , srv1856726378:85 , srv1858934898:85 , srv1900450237:85 , srv1912034675:85 , srv193552618:85 , srv1958662034:85 , srv2023116504:85 , srv2073830444:85 , srv2138767601:85 , srv2142412766:85 , srv225865616:85 , srv286729869:85 , srv292349616:85 , srv330471413:85 , srv358867555:85 , srv42293653:85 , srv43062492:85 , srv492586718:85 , srv597718478:85 , srv629608440:85 , srv686932666:85 , srv771175222:85 , srv784673670:85 , srv841853828:85 , srv884117685:85 , srv899609166:85 , srv919307461:85 , srv933123168:85 , srv935608429:85 , srv964992621:85 , srv986013611:85 , srv1076033619:86 , srv1149741582:630 } 2023-05-19 20:53:29,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-19 20:53:29,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv907942435=93, srv923688601=95, srv2073830444=57, srv899609166=92, srv1559423834=28, srv1795319853=38, srv1389159407=22, srv1133841743=12, srv841853828=89, srv1596708188=30, srv1813841314=39, srv1124417543=10, srv752802284=85, srv315809721=68, srv935608429=97, srv933123168=96, srv263883458=64, srv1120485660=9, srv1055371113=1, srv700132935=83, srv1923383578=48, srv1823245342=40, srv1764481515=37, srv704672012=84, srv2142412766=62, srv1074435512=4, srv330471413=71, srv1075199380=5, srv1076033619=6, srv629608440=80, srv686932666=82, srv20025411=54, srv1399393919=23, srv1459638471=24, srv319734629=70, srv2106068747=59, srv492586718=77, srv771175222=87, srv2138767601=61, srv1722814966=34, srv1186109526=14, srv875956236=90, srv1899623830=45, srv286729869=66, srv354061417=72, srv1749701756=36, srv964992621=98, srv1958662034=51, srv225865616=63, srv1544977718=27, srv759461183=86, srv1473105983=25, srv1900450237=46, srv1723186684=35, srv1388803885=21, srv653500979=81, srv36481867=74, srv43062492=76, srv597718478=79, srv1828536048=41, srv2081523747=58, srv1125463396=11, srv292349616=67, srv919307461=94, srv1091388866=8, srv784673670=88, srv986013611=99, srv1220183021=15, srv1912034675=47, srv1361562873=20, srv193552618=49, srv280558397=65, srv1856726378=43, srv1021095887=0, srv1829759178=42, srv1965876698=53, srv2117366843=60, srv884117685=91, srv1079822256=7, srv42293653=75, srv497211644=78, srv358867555=73, srv1339444637=18, srv156279995=29, srv2023116504=55, srv1257942641=17, srv1701860532=33, srv1149741582=13, srv1609110706=31, srv1661528166=32, srv2040769922=56, srv1543635117=26, srv1065568386=3, srv1255424961=16, srv1961725001=52, srv134982537=19, srv319276580=69, srv1858934898=44, srv1951285527=50, srv1061220919=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-19 20:53:29,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-19 20:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-19 20:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-19 20:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-19 20:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-19 20:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-19 20:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-19 20:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-19 20:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-19 20:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-19 20:53:29,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-19 20:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-19 20:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-19 20:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-19 20:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-19 20:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-19 20:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-19 20:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-19 20:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-19 20:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-19 20:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-19 20:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-19 20:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-19 20:53:29,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-19 20:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-19 20:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-19 20:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-19 20:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-19 20:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-19 20:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-19 20:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-19 20:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-19 20:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-19 20:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-19 20:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-19 20:53:29,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-19 20:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-19 20:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-19 20:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-19 20:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-19 20:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-19 20:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-19 20:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-19 20:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-19 20:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-19 20:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-19 20:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-19 20:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-19 20:53:29,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-19 20:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-19 20:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-19 20:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-19 20:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-19 20:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-19 20:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-19 20:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-19 20:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-19 20:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-19 20:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-19 20:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-19 20:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-19 20:53:29,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-19 20:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-19 20:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-19 20:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-19 20:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-19 20:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-19 20:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-19 20:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-19 20:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-19 20:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-19 20:53:29,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-19 20:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-19 20:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-19 20:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-19 20:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-19 20:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-19 20:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-19 20:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-19 20:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-19 20:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-19 20:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-19 20:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-19 20:53:29,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-19 20:53:29,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-19 20:53:29,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-19 20:53:29,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-19 20:53:29,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-19 20:53:29,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-19 20:53:29,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-19 20:53:29,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-19 20:53:29,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-19 20:53:29,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-19 20:53:29,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-19 20:53:29,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-19 20:53:29,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-19 20:53:29,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-19 20:53:29,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-19 20:53:29,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-19 20:53:29,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-19 20:53:29,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-19 20:53:29,990 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.965681556703379E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-19 20:53:29,991 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1899623830:83 , srv36481867:83 , srv1074435512:84 , srv1075199380:84 , srv1091388866:84 , srv1120485660:84 , srv1124417543:84 , srv1133841743:84 , srv1255424961:84 , srv1339444637:84 , srv134982537:84 , srv1388803885:84 , srv1399393919:84 , srv1473105983:84 , srv1543635117:84 , srv1544977718:84 , srv1559423834:84 , srv156279995:84 , srv1701860532:84 , srv1828536048:84 , srv1923383578:84 , srv1951285527:84 , srv1961725001:84 , srv1965876698:84 , srv20025411:84 , srv2040769922:84 , srv2081523747:84 , srv2106068747:84 , srv2117366843:84 , srv263883458:84 , srv280558397:84 , srv315809721:84 , srv319276580:84 , srv319734629:84 , srv354061417:84 , srv497211644:84 , srv653500979:84 , srv700132935:84 , srv704672012:84 , srv752802284:84 , srv759461183:84 , srv875956236:84 , srv907942435:84 , srv923688601:84 , srv1021095887:85 , srv1055371113:85 , srv1061220919:85 , srv1065568386:85 , srv1079822256:85 , srv1125463396:85 , srv1186109526:85 , srv1220183021:85 , srv1257942641:85 , srv1361562873:85 , srv1389159407:85 , srv1459638471:85 , srv1596708188:85 , srv1609110706:85 , srv1661528166:85 , srv1722814966:85 , srv1723186684:85 , srv1749701756:85 , srv1764481515:85 , srv1795319853:85 , srv1813841314:85 , srv1823245342:85 , srv1829759178:85 , srv1856726378:85 , srv1858934898:85 , srv1900450237:85 , srv1912034675:85 , srv193552618:85 , srv1958662034:85 , srv2023116504:85 , srv2073830444:85 , srv2138767601:85 , srv2142412766:85 , srv225865616:85 , srv286729869:85 , srv292349616:85 , srv330471413:85 , srv358867555:85 , srv42293653:85 , srv43062492:85 , srv492586718:85 , srv597718478:85 , srv629608440:85 , srv686932666:85 , srv771175222:85 , srv784673670:85 , srv841853828:85 , srv884117685:85 , srv899609166:85 , srv919307461:85 , srv933123168:85 , srv935608429:85 , srv964992621:85 , srv986013611:85 , srv1076033619:86 , srv1149741582:630 } 2023-05-19 20:53:30,018 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=423 (was 391) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=8044 (was 8657) 2023-05-19 20:53:30,029 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=423, ProcessCount=172, AvailableMemoryMB=8043 2023-05-19 20:53:30,029 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-19 20:53:30,030 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-19 20:53:30,040 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1002490283:84 , srv1113828861:84 , srv115354420:84 , srv1177162906:84 , srv118739325:84 , srv1188912816:84 , srv1233661814:84 , srv1349475833:84 , srv1478318448:84 , srv1580639808:84 , srv1701486594:84 , srv1721234446:84 , srv1788548722:84 , srv187337574:84 , srv193387463:84 , srv1942859701:84 , srv1979516491:84 , srv202627656:84 , srv265486890:84 , srv290856333:84 , srv349759811:84 , srv378347226:84 , srv392427934:84 , srv67461801:84 , srv694571130:84 , srv757610984:84 , srv882311975:84 , srv893261705:84 , srv932541505:84 , srv2118257623:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-19 20:53:30,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-19 20:53:30,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1113828861=1, srv265486890=19, srv1349475833=7, srv1942859701=15, srv1788548722=12, srv202627656=17, srv67461801=24, srv1580639808=9, srv2118257623=18, srv378347226=22, srv1701486594=10, srv392427934=23, srv1177162906=3, srv118739325=4, srv1188912816=5, srv893261705=28, srv1002490283=0, srv1478318448=8, srv290856333=20, srv187337574=13, srv757610984=26, srv1721234446=11, srv115354420=2, srv932541505=29, srv193387463=14, srv694571130=25, srv1979516491=16, srv882311975=27, srv1233661814=6, srv349759811=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-19 20:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-19 20:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-19 20:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-19 20:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-19 20:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-19 20:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-19 20:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-19 20:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-19 20:53:30,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-19 20:53:30,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-19 20:53:30,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-19 20:53:30,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-19 20:53:30,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-19 20:53:30,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-19 20:53:30,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-19 20:53:30,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-19 20:53:30,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-19 20:53:30,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-19 20:53:30,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-19 20:53:30,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-19 20:53:30,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-19 20:53:30,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-19 20:53:30,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-19 20:53:30,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-19 20:53:30,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-19 20:53:30,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-19 20:53:30,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-19 20:53:30,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-19 20:53:30,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-19 20:53:30,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-19 20:53:30,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-19 20:53:30,068 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0696970118488827, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-19 20:53:30,069 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.99124126562728, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0696970118488827, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-19 20:55:30,055 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6472586 different iterations. Found a solution that moves 1846 regions; Going from a computed imbalance of 0.99124126562728 to a new imbalance of 4.308656090753194E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6837037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-19 20:55:30,062 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1002490283:90 , srv1113828861:90 , srv115354420:90 , srv1177162906:90 , srv118739325:90 , srv1188912816:90 , srv1233661814:90 , srv1349475833:90 , srv1478318448:90 , srv1580639808:90 , srv1701486594:90 , srv1721234446:90 , srv1788548722:90 , srv187337574:90 , srv193387463:90 , srv1942859701:90 , srv1979516491:90 , srv202627656:90 , srv2118257623:90 , srv265486890:90 , srv290856333:90 , srv349759811:90 , srv378347226:90 , srv392427934:90 , srv67461801:90 , srv694571130:90 , srv757610984:90 , srv882311975:90 , srv893261705:90 , srv932541505:90 } 2023-05-19 20:55:30,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-19 20:55:30,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1113828861=1, srv265486890=19, srv1349475833=7, srv1942859701=15, srv1788548722=12, srv202627656=17, srv67461801=24, srv1580639808=9, srv2118257623=18, srv378347226=22, srv1701486594=10, srv392427934=23, srv1177162906=3, srv118739325=4, srv1188912816=5, srv893261705=28, srv1002490283=0, srv1478318448=8, srv290856333=20, srv187337574=13, srv757610984=26, srv1721234446=11, srv115354420=2, srv932541505=29, srv193387463=14, srv694571130=25, srv1979516491=16, srv882311975=27, srv1233661814=6, srv349759811=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-19 20:55:30,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-19 20:55:30,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-19 20:55:30,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-19 20:55:30,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-19 20:55:30,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-19 20:55:30,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-19 20:55:30,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-19 20:55:30,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-19 20:55:30,086 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-19 20:55:30,086 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1002490283:90 , srv1113828861:90 , srv115354420:90 , srv1177162906:90 , srv118739325:90 , srv1188912816:90 , srv1233661814:90 , srv1349475833:90 , srv1478318448:90 , srv1580639808:90 , srv1701486594:90 , srv1721234446:90 , srv1788548722:90 , srv187337574:90 , srv193387463:90 , srv1942859701:90 , srv1979516491:90 , srv202627656:90 , srv2118257623:90 , srv265486890:90 , srv290856333:90 , srv349759811:90 , srv378347226:90 , srv392427934:90 , srv67461801:90 , srv694571130:90 , srv757610984:90 , srv882311975:90 , srv893261705:90 , srv932541505:90 } 2023-05-19 20:55:30,101 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=229 (was 423), ProcessCount=172 (was 172), AvailableMemoryMB=7475 (was 8043)