2023-10-22 05:53:29,925 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-22 05:53:30,260 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-22 05:53:30,400 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-22 05:53:30,493 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-22 05:53:30,503 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-22 05:53:30,521 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=405, ProcessCount=172, AvailableMemoryMB=8775 2023-10-22 05:53:30,524 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-22 05:53:30,524 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-22 05:53:30,645 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013462856:84 , srv1026397040:84 , srv1098929659:84 , srv1122416143:84 , srv1125216336:84 , srv117620623:84 , srv1184657828:84 , srv1201422454:84 , srv1231114039:84 , srv1246642459:84 , srv1261540878:84 , srv1298902467:84 , srv1299528281:84 , srv1311513619:84 , srv1318240473:84 , srv133348285:84 , srv1376132337:84 , srv1383409674:84 , srv140891687:84 , srv144294765:84 , srv1581255797:84 , srv1585849778:84 , srv1587075134:84 , srv1602473041:84 , srv1627215862:84 , srv1645054705:84 , srv1673412628:84 , srv1681788727:84 , srv1715754948:84 , srv1759625410:84 , srv1764733675:84 , srv1764950590:84 , srv1817051675:84 , srv1839380450:84 , srv1889546448:84 , srv1890334225:84 , srv1930285747:84 , srv1933428669:84 , srv1933441394:84 , srv19548311:84 , srv1958087025:84 , srv198294120:84 , srv1999155523:84 , srv2002313204:84 , srv2022856727:84 , srv2035032832:84 , srv2062921597:84 , srv2083689107:84 , srv2103032505:84 , srv217935322:84 , srv23063168:84 , srv257951261:84 , srv277177157:84 , srv28309043:84 , srv285168293:84 , srv299996222:84 , srv300820540:84 , srv303933372:84 , srv30648072:84 , srv314381580:84 , srv323447910:84 , srv326826729:84 , srv341534540:84 , srv352649824:84 , srv356357786:84 , srv371491542:84 , srv37642435:84 , srv407116648:84 , srv444926845:84 , srv448005075:84 , srv450835751:84 , srv489388751:84 , srv493128391:84 , srv523143338:84 , srv538219210:84 , srv548458538:84 , srv561458825:84 , srv568072053:84 , srv577872629:84 , srv581208843:84 , srv58696394:84 , srv597496948:84 , srv599602867:84 , srv627736328:84 , srv734260795:84 , srv73673575:84 , srv75783808:84 , srv758295194:84 , srv766715313:84 , srv773215167:84 , srv792938805:84 , srv82042062:84 , srv829793925:84 , srv925322750:84 , srv936395698:84 , srv937918212:84 , srv946652972:84 , srv963156765:84 , srv963297726:84 , srv1637945578:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-22 05:53:30,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-22 05:53:30,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1026397040=1, srv1122416143=3, srv2083689107=48, srv766715313=89, srv523143338=74, srv450835751=71, srv1261540878=10, srv581208843=80, srv1231114039=8, srv1673412628=27, srv2035032832=46, srv1585849778=21, srv1298902467=11, srv561458825=77, srv937918212=96, srv577872629=79, srv2062921597=47, srv1999155523=43, srv314381580=60, srv323447910=61, srv407116648=68, srv326826729=62, srv963156765=98, srv30648072=59, srv1764950590=32, srv489388751=72, srv548458538=76, srv936395698=95, srv144294765=19, srv1681788727=28, srv1246642459=9, srv1627215862=24, srv37642435=67, srv303933372=58, srv1645054705=26, srv19548311=40, srv734260795=85, srv627736328=84, srv117620623=5, srv1958087025=41, srv356357786=65, srv2002313204=44, srv28309043=54, srv58696394=81, srv277177157=53, srv1839380450=34, srv1890334225=36, srv341534540=63, srv597496948=82, srv493128391=73, srv538219210=75, srv448005075=70, srv133348285=15, srv1933428669=38, srv1299528281=12, srv352649824=64, srv599602867=83, srv1602473041=23, srv299996222=56, srv371491542=66, srv773215167=90, srv1311513619=13, srv198294120=42, srv217935322=50, srv963297726=99, srv1098929659=2, srv73673575=86, srv1318240473=14, srv1817051675=33, srv1637945578=25, srv140891687=18, srv1125216336=4, srv300820540=57, srv758295194=88, srv1759625410=30, srv1587075134=22, srv257951261=52, srv1184657828=6, srv568072053=78, srv1764733675=31, srv1581255797=20, srv1930285747=37, srv23063168=51, srv1201422454=7, srv1889546448=35, srv792938805=91, srv925322750=94, srv1383409674=17, srv1013462856=0, srv285168293=55, srv1933441394=39, srv2022856727=45, srv75783808=87, srv1715754948=29, srv946652972=97, srv1376132337=16, srv829793925=93, srv2103032505=49, srv444926845=69, srv82042062=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-22 05:53:30,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-22 05:53:30,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-22 05:53:30,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-22 05:53:30,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-22 05:53:30,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-22 05:53:30,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-22 05:53:30,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-22 05:53:30,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-22 05:53:30,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-22 05:53:30,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-22 05:53:30,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-22 05:53:30,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-22 05:53:30,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-22 05:53:30,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-22 05:53:30,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-22 05:53:30,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-22 05:53:30,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-22 05:53:30,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-22 05:53:30,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-22 05:53:30,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-22 05:53:30,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-22 05:53:30,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-22 05:53:30,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-22 05:53:30,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-22 05:53:30,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-22 05:53:30,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-22 05:53:30,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-22 05:53:30,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-22 05:53:30,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-22 05:53:30,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-22 05:53:30,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-22 05:53:30,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-22 05:53:30,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-22 05:53:30,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-22 05:53:30,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-22 05:53:30,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-22 05:53:30,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-22 05:53:30,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-22 05:53:30,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-22 05:53:30,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-22 05:53:30,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-22 05:53:30,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-22 05:53:30,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-22 05:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-22 05:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-22 05:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-22 05:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-22 05:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-22 05:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-22 05:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-22 05:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-22 05:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-22 05:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-22 05:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-22 05:53:30,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-22 05:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-22 05:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-22 05:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-22 05:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-22 05:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-22 05:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-22 05:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-22 05:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-22 05:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-22 05:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-22 05:53:30,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-22 05:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-22 05:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-22 05:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-22 05:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-22 05:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-22 05:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-22 05:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-22 05:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-22 05:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-22 05:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-22 05:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-22 05:53:30,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-22 05:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-22 05:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-22 05:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-22 05:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-22 05:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-22 05:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-22 05:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-22 05:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-22 05:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-22 05:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-22 05:53:30,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-22 05:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-22 05:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-22 05:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-22 05:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-22 05:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-22 05:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-22 05:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-22 05:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-22 05:53:30,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-22 05:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-22 05:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-22 05:53:30,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-22 05:53:30,915 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-22 05:53:30,916 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-22 05:53:30,916 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-22 05:53:31,264 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 500 ms to try 5000 different iterations. Found a solution that moves 4651 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18513317480263472. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038356626895314744, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05813216877096288, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5167777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038356626895314744, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18066666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-22 05:53:31,288 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv946652972:82 , srv1122416143:83 , srv1201422454:83 , srv2062921597:83 , srv323447910:83 , srv792938805:83 , srv1933428669:84 , srv19548311:84 , srv198294120:84 , srv2083689107:84 , srv217935322:84 , srv1246642459:85 , srv1299528281:85 , srv133348285:85 , srv1645054705:85 , srv1681788727:85 , srv1759625410:85 , srv1817051675:85 , srv1933441394:85 , srv2002313204:85 , srv2103032505:85 , srv285168293:85 , srv303933372:85 , srv30648072:85 , srv444926845:85 , srv489388751:85 , srv548458538:85 , srv561458825:85 , srv597496948:85 , srv73673575:85 , srv75783808:85 , srv925322750:85 , srv963297726:85 , srv1013462856:86 , srv1098929659:86 , srv1125216336:86 , srv1261540878:86 , srv1311513619:86 , srv1581255797:86 , srv1587075134:86 , srv1602473041:86 , srv1627215862:86 , srv1673412628:86 , srv1764733675:86 , srv1890334225:86 , srv1930285747:86 , srv2035032832:86 , srv23063168:86 , srv257951261:86 , srv299996222:86 , srv326826729:86 , srv341534540:86 , srv352649824:86 , srv37642435:86 , srv448005075:86 , srv450835751:86 , srv58696394:86 , srv599602867:86 , srv937918212:86 , srv117620623:87 , srv1231114039:87 , srv1383409674:87 , srv140891687:87 , srv1889546448:87 , srv2022856727:87 , srv28309043:87 , srv523143338:87 , srv577872629:87 , srv734260795:87 , srv758295194:87 , srv766715313:87 , srv1184657828:88 , srv1298902467:88 , srv1318240473:88 , srv1376132337:88 , srv1585849778:88 , srv1764950590:88 , srv1839380450:88 , srv1958087025:88 , srv1999155523:88 , srv356357786:88 , srv371491542:88 , srv568072053:88 , srv581208843:88 , srv773215167:88 , srv829793925:88 , srv963156765:88 , srv1026397040:89 , srv277177157:89 , srv314381580:89 , srv936395698:89 , srv538219210:90 , srv82042062:90 , srv627736328:91 , srv144294765:92 , srv493128391:92 , srv1715754948:94 , srv407116648:94 , srv300820540:95 , srv1637945578:431 } 2023-10-22 05:53:31,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-22 05:53:31,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1026397040=1, srv1122416143=3, srv2083689107=48, srv766715313=89, srv523143338=74, srv450835751=71, srv1261540878=10, srv581208843=80, srv1231114039=8, srv1673412628=27, srv2035032832=46, srv1585849778=21, srv1298902467=11, srv561458825=77, srv937918212=96, srv577872629=79, srv2062921597=47, srv1999155523=43, srv314381580=60, srv323447910=61, srv407116648=68, srv326826729=62, srv963156765=98, srv30648072=59, srv1764950590=32, srv489388751=72, srv548458538=76, srv936395698=95, srv144294765=19, srv1681788727=28, srv1246642459=9, srv1627215862=24, srv37642435=67, srv303933372=58, srv1645054705=26, srv19548311=40, srv734260795=85, srv627736328=84, srv117620623=5, srv1958087025=41, srv356357786=65, srv2002313204=44, srv28309043=54, srv58696394=81, srv277177157=53, srv1839380450=34, srv1890334225=36, srv341534540=63, srv597496948=82, srv493128391=73, srv538219210=75, srv448005075=70, srv133348285=15, srv1933428669=38, srv1299528281=12, srv352649824=64, srv599602867=83, srv1602473041=23, srv299996222=56, srv371491542=66, srv773215167=90, srv1311513619=13, srv198294120=42, srv217935322=50, srv963297726=99, srv1098929659=2, srv73673575=86, srv1318240473=14, srv1817051675=33, srv1637945578=25, srv140891687=18, srv1125216336=4, srv300820540=57, srv758295194=88, srv1759625410=30, srv1587075134=22, srv257951261=52, srv1184657828=6, srv568072053=78, srv1764733675=31, srv1581255797=20, srv1930285747=37, srv23063168=51, srv1201422454=7, srv1889546448=35, srv792938805=91, srv925322750=94, srv1383409674=17, srv1013462856=0, srv285168293=55, srv1933441394=39, srv2022856727=45, srv75783808=87, srv1715754948=29, srv946652972=97, srv1376132337=16, srv829793925=93, srv2103032505=49, srv444926845=69, srv82042062=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-22 05:53:31,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-22 05:53:31,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-22 05:53:31,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-22 05:53:31,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-22 05:53:31,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-22 05:53:31,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-22 05:53:31,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-22 05:53:31,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-22 05:53:31,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-22 05:53:31,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-22 05:53:31,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-22 05:53:31,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-22 05:53:31,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-22 05:53:31,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-22 05:53:31,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-22 05:53:31,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-22 05:53:31,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-22 05:53:31,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-22 05:53:31,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-22 05:53:31,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-22 05:53:31,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-22 05:53:31,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-22 05:53:31,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-22 05:53:31,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-22 05:53:31,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-22 05:53:31,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-22 05:53:31,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-22 05:53:31,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-22 05:53:31,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-22 05:53:31,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-22 05:53:31,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-22 05:53:31,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-22 05:53:31,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-22 05:53:31,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-22 05:53:31,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-22 05:53:31,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-22 05:53:31,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-22 05:53:31,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-22 05:53:31,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-22 05:53:31,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-22 05:53:31,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-22 05:53:31,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-22 05:53:31,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-22 05:53:31,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-22 05:53:31,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-22 05:53:31,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-22 05:53:31,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-22 05:53:31,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-22 05:53:31,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-22 05:53:31,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-22 05:53:31,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-22 05:53:31,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-22 05:53:31,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-22 05:53:31,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-22 05:53:31,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-22 05:53:31,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-22 05:53:31,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-22 05:53:31,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-22 05:53:31,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-22 05:53:31,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-22 05:53:31,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-22 05:53:31,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-22 05:53:31,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-22 05:53:31,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-22 05:53:31,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-22 05:53:31,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-22 05:53:31,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-22 05:53:31,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-22 05:53:31,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-22 05:53:31,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-22 05:53:31,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-22 05:53:31,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-22 05:53:31,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-22 05:53:31,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-22 05:53:31,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-22 05:53:31,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-22 05:53:31,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-22 05:53:31,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-22 05:53:31,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-22 05:53:31,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-22 05:53:31,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-22 05:53:31,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-22 05:53:31,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-22 05:53:31,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-22 05:53:31,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-22 05:53:31,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-22 05:53:31,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-22 05:53:31,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-22 05:53:31,475 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038356626895314744, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05813216877096288, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038356626895314744, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18066666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-22 05:53:31,475 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-22 05:53:31,475 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18510060780456628, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038356626895314744, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05813216877096288, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038356626895314744, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18066666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-22 05:53:31,726 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 378 ms to try 5000 different iterations. Found a solution that moves 2673 regions; Going from a computed imbalance of 0.18510060780456628 to a new imbalance of 0.014586284053663884. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021386064646574523); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03336025848581182, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.297, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021386064646574523); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-22 05:53:31,738 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv300820540:76 , srv144294765:79 , srv1715754948:79 , srv627736328:79 , srv407116648:80 , srv493128391:80 , srv1376132337:82 , srv277177157:82 , srv963156765:82 , srv1026397040:83 , srv1184657828:83 , srv1298902467:83 , srv1585849778:83 , srv1958087025:83 , srv2022856727:83 , srv82042062:83 , srv829793925:83 , srv1261540878:84 , srv1311513619:84 , srv1318240473:84 , srv1587075134:84 , srv1999155523:84 , srv257951261:84 , srv314381580:84 , srv371491542:84 , srv37642435:84 , srv523143338:84 , srv538219210:84 , srv577872629:84 , srv766715313:84 , srv773215167:84 , srv936395698:84 , srv133348285:85 , srv140891687:85 , srv1645054705:85 , srv1764950590:85 , srv1839380450:85 , srv1933441394:85 , srv23063168:85 , srv28309043:85 , srv285168293:85 , srv299996222:85 , srv450835751:85 , srv548458538:85 , srv568072053:85 , srv581208843:85 , srv758295194:85 , srv1013462856:86 , srv1246642459:86 , srv1299528281:86 , srv1383409674:86 , srv1581255797:86 , srv1602473041:86 , srv1673412628:86 , srv1764733675:86 , srv1817051675:86 , srv1890334225:86 , srv1930285747:86 , srv198294120:86 , srv2035032832:86 , srv217935322:86 , srv356357786:86 , srv561458825:86 , srv58696394:86 , srv1098929659:87 , srv1125216336:87 , srv117620623:87 , srv1933428669:87 , srv2002313204:87 , srv326826729:87 , srv341534540:87 , srv352649824:87 , srv597496948:87 , srv599602867:87 , srv734260795:87 , srv73673575:87 , srv925322750:87 , srv937918212:87 , srv1231114039:88 , srv1627215862:88 , srv1681788727:88 , srv1759625410:88 , srv2103032505:88 , srv303933372:88 , srv30648072:88 , srv444926845:88 , srv489388751:88 , srv75783808:88 , srv1122416143:89 , srv1201422454:89 , srv19548311:89 , srv323447910:89 , srv448005075:89 , srv963297726:89 , srv946652972:90 , srv1889546448:91 , srv2083689107:91 , srv792938805:91 , srv2062921597:92 , srv1637945578:533 } 2023-10-22 05:53:31,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-22 05:53:31,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1026397040=1, srv1122416143=3, srv2083689107=48, srv766715313=89, srv523143338=74, srv450835751=71, srv1261540878=10, srv581208843=80, srv1231114039=8, srv1673412628=27, srv2035032832=46, srv1585849778=21, srv1298902467=11, srv561458825=77, srv937918212=96, srv577872629=79, srv2062921597=47, srv1999155523=43, srv314381580=60, srv323447910=61, srv407116648=68, srv326826729=62, srv963156765=98, srv30648072=59, srv1764950590=32, srv489388751=72, srv548458538=76, srv936395698=95, srv144294765=19, srv1681788727=28, srv1246642459=9, srv1627215862=24, srv37642435=67, srv303933372=58, srv1645054705=26, srv19548311=40, srv734260795=85, srv627736328=84, srv117620623=5, srv1958087025=41, srv356357786=65, srv2002313204=44, srv28309043=54, srv58696394=81, srv277177157=53, srv1839380450=34, srv1890334225=36, srv341534540=63, srv597496948=82, srv493128391=73, srv538219210=75, srv448005075=70, srv133348285=15, srv1933428669=38, srv1299528281=12, srv352649824=64, srv599602867=83, srv1602473041=23, srv299996222=56, srv371491542=66, srv773215167=90, srv1311513619=13, srv198294120=42, srv217935322=50, srv963297726=99, srv1098929659=2, srv73673575=86, srv1318240473=14, srv1817051675=33, srv1637945578=25, srv140891687=18, srv1125216336=4, srv300820540=57, srv758295194=88, srv1759625410=30, srv1587075134=22, srv257951261=52, srv1184657828=6, srv568072053=78, srv1764733675=31, srv1581255797=20, srv1930285747=37, srv23063168=51, srv1201422454=7, srv1889546448=35, srv792938805=91, srv925322750=94, srv1383409674=17, srv1013462856=0, srv285168293=55, srv1933441394=39, srv2022856727=45, srv75783808=87, srv1715754948=29, srv946652972=97, srv1376132337=16, srv829793925=93, srv2103032505=49, srv444926845=69, srv82042062=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-22 05:53:31,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-22 05:53:31,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-22 05:53:31,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-22 05:53:31,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-22 05:53:31,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-22 05:53:31,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-22 05:53:31,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-22 05:53:31,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-22 05:53:31,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-22 05:53:31,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-22 05:53:31,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-22 05:53:31,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-22 05:53:31,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-22 05:53:31,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-22 05:53:31,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-22 05:53:31,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-22 05:53:31,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-22 05:53:31,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-22 05:53:31,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-22 05:53:31,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-22 05:53:31,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-22 05:53:31,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-22 05:53:31,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-22 05:53:31,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-22 05:53:31,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-22 05:53:31,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-22 05:53:31,879 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021386064646574523); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03336025848581182, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021386064646574523); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-22 05:53:31,880 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-22 05:53:31,880 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01456756730762285, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021386064646574523); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03336025848581182, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021386064646574523); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-22 05:53:32,153 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 372 ms to try 5000 different iterations. Found a solution that moves 956 regions; Going from a computed imbalance of 0.01456756730762285 to a new imbalance of 0.0010812330232515192. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006089903001559224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008863600391305173); MoveCostFunction : (multiplier=7.0, imbalance=0.10622222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006089903001559224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-22 05:53:32,155 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1889546448:80 , srv1231114039:82 , srv117620623:83 , srv2062921597:83 , srv2083689107:83 , srv448005075:83 , srv538219210:83 , srv734260795:83 , srv1383409674:84 , srv1627215862:84 , srv1681788727:84 , srv1715754948:84 , srv19548311:84 , srv2103032505:84 , srv303933372:84 , srv314381580:84 , srv326826729:84 , srv352649824:84 , srv356357786:84 , srv568072053:84 , srv599602867:84 , srv792938805:84 , srv963297726:84 , srv1013462856:85 , srv1026397040:85 , srv1098929659:85 , srv1125216336:85 , srv1201422454:85 , srv140891687:85 , srv1581255797:85 , srv1673412628:85 , srv1759625410:85 , srv1764950590:85 , srv1839380450:85 , srv1890334225:85 , srv1930285747:85 , srv1999155523:85 , srv2002313204:85 , srv30648072:85 , srv323447910:85 , srv341534540:85 , srv371491542:85 , srv407116648:85 , srv444926845:85 , srv489388751:85 , srv493128391:85 , srv581208843:85 , srv58696394:85 , srv73673575:85 , srv75783808:85 , srv758295194:85 , srv773215167:85 , srv82042062:85 , srv925322750:85 , srv936395698:85 , srv937918212:85 , srv1122416143:86 , srv1184657828:86 , srv1318240473:86 , srv144294765:86 , srv1585849778:86 , srv1602473041:86 , srv1764733675:86 , srv1933428669:86 , srv1958087025:86 , srv2035032832:86 , srv277177157:86 , srv28309043:86 , srv299996222:86 , srv300820540:86 , srv450835751:86 , srv523143338:86 , srv561458825:86 , srv577872629:86 , srv766715313:86 , srv946652972:86 , srv1246642459:87 , srv1298902467:87 , srv1299528281:87 , srv1311513619:87 , srv133348285:87 , srv1587075134:87 , srv1817051675:87 , srv1933441394:87 , srv198294120:87 , srv2022856727:87 , srv217935322:87 , srv23063168:87 , srv37642435:87 , srv548458538:87 , srv597496948:87 , srv627736328:87 , srv829793925:87 , srv1261540878:88 , srv1376132337:88 , srv1645054705:88 , srv257951261:88 , srv285168293:88 , srv963156765:88 , srv1637945578:548 } 2023-10-22 05:53:32,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-22 05:53:32,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1026397040=1, srv1122416143=3, srv2083689107=48, srv766715313=89, srv523143338=74, srv450835751=71, srv1261540878=10, srv581208843=80, srv1231114039=8, srv1673412628=27, srv2035032832=46, srv1585849778=21, srv1298902467=11, srv561458825=77, srv937918212=96, srv577872629=79, srv2062921597=47, srv1999155523=43, srv314381580=60, srv323447910=61, srv407116648=68, srv326826729=62, srv963156765=98, srv30648072=59, srv1764950590=32, srv489388751=72, srv548458538=76, srv936395698=95, srv144294765=19, srv1681788727=28, srv1246642459=9, srv1627215862=24, srv37642435=67, srv303933372=58, srv1645054705=26, srv19548311=40, srv734260795=85, srv627736328=84, srv117620623=5, srv1958087025=41, srv356357786=65, srv2002313204=44, srv28309043=54, srv58696394=81, srv277177157=53, srv1839380450=34, srv1890334225=36, srv341534540=63, srv597496948=82, srv493128391=73, srv538219210=75, srv448005075=70, srv133348285=15, srv1933428669=38, srv1299528281=12, srv352649824=64, srv599602867=83, srv1602473041=23, srv299996222=56, srv371491542=66, srv773215167=90, srv1311513619=13, srv198294120=42, srv217935322=50, srv963297726=99, srv1098929659=2, srv73673575=86, srv1318240473=14, srv1817051675=33, srv1637945578=25, srv140891687=18, srv1125216336=4, srv300820540=57, srv758295194=88, srv1759625410=30, srv1587075134=22, srv257951261=52, srv1184657828=6, srv568072053=78, srv1764733675=31, srv1581255797=20, srv1930285747=37, srv23063168=51, srv1201422454=7, srv1889546448=35, srv792938805=91, srv925322750=94, srv1383409674=17, srv1013462856=0, srv285168293=55, srv1933441394=39, srv2022856727=45, srv75783808=87, srv1715754948=29, srv946652972=97, srv1376132337=16, srv829793925=93, srv2103032505=49, srv444926845=69, srv82042062=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-22 05:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-22 05:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-22 05:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-22 05:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-22 05:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-22 05:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-22 05:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-22 05:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-22 05:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-22 05:53:32,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-22 05:53:32,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-22 05:53:32,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-22 05:53:32,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-22 05:53:32,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-22 05:53:32,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-22 05:53:32,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-22 05:53:32,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-22 05:53:32,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-22 05:53:32,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-22 05:53:32,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-22 05:53:32,302 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006089903001559224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008863600391305173); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006089903001559224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-22 05:53:32,303 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-22 05:53:32,303 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.001074538968176611, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006089903001559224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008863600391305173); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006089903001559224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-22 05:53:32,706 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 519 ms to try 5000 different iterations. Found a solution that moves 270 regions; Going from a computed imbalance of 0.001074538968176611 to a new imbalance of 1.2432292617286242E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.03, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-22 05:53:32,707 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2035032832:83 , srv407116648:83 , srv597496948:83 , srv1098929659:84 , srv1122416143:84 , srv1125216336:84 , srv1246642459:84 , srv1261540878:84 , srv1298902467:84 , srv1299528281:84 , srv1318240473:84 , srv1376132337:84 , srv1585849778:84 , srv1602473041:84 , srv1627215862:84 , srv1645054705:84 , srv1759625410:84 , srv1764733675:84 , srv1764950590:84 , srv1817051675:84 , srv1839380450:84 , srv1889546448:84 , srv1930285747:84 , srv19548311:84 , srv2022856727:84 , srv2062921597:84 , srv2083689107:84 , srv23063168:84 , srv28309043:84 , srv285168293:84 , srv30648072:84 , srv323447910:84 , srv341534540:84 , srv444926845:84 , srv448005075:84 , srv450835751:84 , srv489388751:84 , srv581208843:84 , srv58696394:84 , srv599602867:84 , srv75783808:84 , srv792938805:84 , srv82042062:84 , srv829793925:84 , srv937918212:84 , srv946652972:84 , srv963156765:84 , srv963297726:84 , srv1013462856:85 , srv1026397040:85 , srv117620623:85 , srv1201422454:85 , srv1231114039:85 , srv1311513619:85 , srv133348285:85 , srv1383409674:85 , srv140891687:85 , srv144294765:85 , srv1581255797:85 , srv1673412628:85 , srv1681788727:85 , srv1715754948:85 , srv1890334225:85 , srv1933428669:85 , srv1933441394:85 , srv1958087025:85 , srv198294120:85 , srv1999155523:85 , srv2002313204:85 , srv2103032505:85 , srv217935322:85 , srv257951261:85 , srv277177157:85 , srv299996222:85 , srv300820540:85 , srv303933372:85 , srv314381580:85 , srv326826729:85 , srv352649824:85 , srv356357786:85 , srv371491542:85 , srv37642435:85 , srv493128391:85 , srv523143338:85 , srv538219210:85 , srv548458538:85 , srv561458825:85 , srv577872629:85 , srv627736328:85 , srv734260795:85 , srv73673575:85 , srv758295194:85 , srv925322750:85 , srv936395698:85 , srv1184657828:86 , srv1587075134:86 , srv568072053:86 , srv766715313:86 , srv773215167:86 , srv1637945578:631 } 2023-10-22 05:53:32,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-22 05:53:32,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1026397040=1, srv1122416143=3, srv2083689107=48, srv766715313=89, srv523143338=74, srv450835751=71, srv1261540878=10, srv581208843=80, srv1231114039=8, srv1673412628=27, srv2035032832=46, srv1585849778=21, srv1298902467=11, srv561458825=77, srv937918212=96, srv577872629=79, srv2062921597=47, srv1999155523=43, srv314381580=60, srv323447910=61, srv407116648=68, srv326826729=62, srv963156765=98, srv30648072=59, srv1764950590=32, srv489388751=72, srv548458538=76, srv936395698=95, srv144294765=19, srv1681788727=28, srv1246642459=9, srv1627215862=24, srv37642435=67, srv303933372=58, srv1645054705=26, srv19548311=40, srv734260795=85, srv627736328=84, srv117620623=5, srv1958087025=41, srv356357786=65, srv2002313204=44, srv28309043=54, srv58696394=81, srv277177157=53, srv1839380450=34, srv1890334225=36, srv341534540=63, srv597496948=82, srv493128391=73, srv538219210=75, srv448005075=70, srv133348285=15, srv1933428669=38, srv1299528281=12, srv352649824=64, srv599602867=83, srv1602473041=23, srv299996222=56, srv371491542=66, srv773215167=90, srv1311513619=13, srv198294120=42, srv217935322=50, srv963297726=99, srv1098929659=2, srv73673575=86, srv1318240473=14, srv1817051675=33, srv1637945578=25, srv140891687=18, srv1125216336=4, srv300820540=57, srv758295194=88, srv1759625410=30, srv1587075134=22, srv257951261=52, srv1184657828=6, srv568072053=78, srv1764733675=31, srv1581255797=20, srv1930285747=37, srv23063168=51, srv1201422454=7, srv1889546448=35, srv792938805=91, srv925322750=94, srv1383409674=17, srv1013462856=0, srv285168293=55, srv1933441394=39, srv2022856727=45, srv75783808=87, srv1715754948=29, srv946652972=97, srv1376132337=16, srv829793925=93, srv2103032505=49, srv444926845=69, srv82042062=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-22 05:53:32,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-22 05:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-22 05:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-22 05:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-22 05:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-22 05:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-22 05:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-22 05:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-22 05:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-22 05:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-22 05:53:32,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-22 05:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-22 05:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-22 05:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-22 05:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-22 05:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-22 05:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-22 05:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-22 05:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-22 05:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-22 05:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-22 05:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-22 05:53:32,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-22 05:53:32,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-22 05:53:32,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-22 05:53:32,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-22 05:53:32,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-22 05:53:32,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-22 05:53:32,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-22 05:53:32,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-22 05:53:32,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-22 05:53:32,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-22 05:53:32,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-22 05:53:32,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-22 05:53:32,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-22 05:53:32,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-22 05:53:32,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-22 05:53:32,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-22 05:53:32,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-22 05:53:32,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-22 05:53:32,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-22 05:53:32,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-22 05:53:32,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-22 05:53:32,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-22 05:53:32,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-22 05:53:32,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-22 05:53:32,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-22 05:53:32,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-22 05:53:32,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-22 05:53:32,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-22 05:53:32,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-22 05:53:32,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-22 05:53:32,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-22 05:53:32,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-22 05:53:32,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-22 05:53:32,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-22 05:53:32,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-22 05:53:32,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-22 05:53:32,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-22 05:53:32,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-22 05:53:32,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-22 05:53:32,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-22 05:53:32,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-22 05:53:32,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-22 05:53:32,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-22 05:53:32,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-22 05:53:32,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-22 05:53:32,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-22 05:53:32,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-22 05:53:32,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-22 05:53:32,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-22 05:53:32,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-22 05:53:32,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-22 05:53:32,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-22 05:53:32,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-22 05:53:32,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-22 05:53:32,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-22 05:53:32,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-22 05:53:32,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-22 05:53:32,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-22 05:53:32,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-22 05:53:32,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-22 05:53:32,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-22 05:53:32,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-22 05:53:32,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-22 05:53:32,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-22 05:53:32,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-22 05:53:32,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-22 05:53:32,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-22 05:53:32,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-22 05:53:32,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-22 05:53:32,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-22 05:53:32,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-22 05:53:32,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-22 05:53:32,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-22 05:53:32,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-22 05:53:32,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-22 05:53:32,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-22 05:53:32,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-22 05:53:32,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-22 05:53:32,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-22 05:53:32,883 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.0541712209100929E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-22 05:53:32,883 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv2035032832:83 , srv407116648:83 , srv597496948:83 , srv1098929659:84 , srv1122416143:84 , srv1125216336:84 , srv1246642459:84 , srv1261540878:84 , srv1298902467:84 , srv1299528281:84 , srv1318240473:84 , srv1376132337:84 , srv1585849778:84 , srv1602473041:84 , srv1627215862:84 , srv1645054705:84 , srv1759625410:84 , srv1764733675:84 , srv1764950590:84 , srv1817051675:84 , srv1839380450:84 , srv1889546448:84 , srv1930285747:84 , srv19548311:84 , srv2022856727:84 , srv2062921597:84 , srv2083689107:84 , srv23063168:84 , srv28309043:84 , srv285168293:84 , srv30648072:84 , srv323447910:84 , srv341534540:84 , srv444926845:84 , srv448005075:84 , srv450835751:84 , srv489388751:84 , srv581208843:84 , srv58696394:84 , srv599602867:84 , srv75783808:84 , srv792938805:84 , srv82042062:84 , srv829793925:84 , srv937918212:84 , srv946652972:84 , srv963156765:84 , srv963297726:84 , srv1013462856:85 , srv1026397040:85 , srv117620623:85 , srv1201422454:85 , srv1231114039:85 , srv1311513619:85 , srv133348285:85 , srv1383409674:85 , srv140891687:85 , srv144294765:85 , srv1581255797:85 , srv1673412628:85 , srv1681788727:85 , srv1715754948:85 , srv1890334225:85 , srv1933428669:85 , srv1933441394:85 , srv1958087025:85 , srv198294120:85 , srv1999155523:85 , srv2002313204:85 , srv2103032505:85 , srv217935322:85 , srv257951261:85 , srv277177157:85 , srv299996222:85 , srv300820540:85 , srv303933372:85 , srv314381580:85 , srv326826729:85 , srv352649824:85 , srv356357786:85 , srv371491542:85 , srv37642435:85 , srv493128391:85 , srv523143338:85 , srv538219210:85 , srv548458538:85 , srv561458825:85 , srv577872629:85 , srv627736328:85 , srv734260795:85 , srv73673575:85 , srv758295194:85 , srv925322750:85 , srv936395698:85 , srv1184657828:86 , srv1587075134:86 , srv568072053:86 , srv766715313:86 , srv773215167:86 , srv1637945578:631 } 2023-10-22 05:53:32,917 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=405 (was 405), ProcessCount=172 (was 172), AvailableMemoryMB=8275 (was 8775) 2023-10-22 05:53:32,927 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=405, ProcessCount=172, AvailableMemoryMB=8274 2023-10-22 05:53:32,928 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-22 05:53:32,928 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-22 05:53:32,939 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1182921277:84 , srv1350285687:84 , srv138767701:84 , srv1428612270:84 , srv1461431:84 , srv152983832:84 , srv161798207:84 , srv1621437292:84 , srv1630429571:84 , srv163944081:84 , srv1958090542:84 , srv2071952832:84 , srv20902685:84 , srv2111316196:84 , srv2136561743:84 , srv2145918736:84 , srv279001139:84 , srv33233295:84 , srv388135465:84 , srv412402871:84 , srv635101089:84 , srv690448789:84 , srv71873275:84 , srv725032310:84 , srv756175702:84 , srv796216470:84 , srv797373847:84 , srv934883970:84 , srv970295007:84 , srv1256262313:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-22 05:53:32,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-22 05:53:32,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv138767701=3, srv1461431=5, srv152983832=6, srv1958090542=11, srv1350285687=2, srv163944081=10, srv635101089=21, srv71873275=23, srv1630429571=9, srv756175702=25, srv2071952832=12, srv796216470=26, srv388135465=19, srv690448789=22, srv934883970=28, srv2111316196=14, srv1621437292=8, srv2136561743=15, srv725032310=24, srv279001139=17, srv1428612270=4, srv20902685=13, srv970295007=29, srv1182921277=0, srv412402871=20, srv797373847=27, srv161798207=7, srv2145918736=16, srv33233295=18, srv1256262313=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-22 05:53:32,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-22 05:53:32,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-22 05:53:32,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-22 05:53:32,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-22 05:53:32,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-22 05:53:32,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-22 05:53:32,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-22 05:53:32,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-22 05:53:32,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-22 05:53:32,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-22 05:53:32,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-22 05:53:32,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-22 05:53:32,964 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0991029349293573, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-22 05:53:32,964 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912505313376234, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0991029349293573, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-22 05:55:32,953 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5703724 different iterations. Found a solution that moves 1864 regions; Going from a computed imbalance of 0.9912505313376234 to a new imbalance of 4.3506689887128686E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6903703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-22 05:55:32,959 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1182921277:90 , srv1256262313:90 , srv1350285687:90 , srv138767701:90 , srv1428612270:90 , srv1461431:90 , srv152983832:90 , srv161798207:90 , srv1621437292:90 , srv1630429571:90 , srv163944081:90 , srv1958090542:90 , srv2071952832:90 , srv20902685:90 , srv2111316196:90 , srv2136561743:90 , srv2145918736:90 , srv279001139:90 , srv33233295:90 , srv388135465:90 , srv412402871:90 , srv635101089:90 , srv690448789:90 , srv71873275:90 , srv725032310:90 , srv756175702:90 , srv796216470:90 , srv797373847:90 , srv934883970:90 , srv970295007:90 } 2023-10-22 05:55:32,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-22 05:55:32,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv138767701=3, srv1461431=5, srv152983832=6, srv1958090542=11, srv1350285687=2, srv163944081=10, srv635101089=21, srv71873275=23, srv1630429571=9, srv756175702=25, srv2071952832=12, srv796216470=26, srv388135465=19, srv690448789=22, srv934883970=28, srv2111316196=14, srv1621437292=8, srv2136561743=15, srv725032310=24, srv279001139=17, srv1428612270=4, srv20902685=13, srv970295007=29, srv1182921277=0, srv412402871=20, srv797373847=27, srv161798207=7, srv2145918736=16, srv33233295=18, srv1256262313=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-22 05:55:32,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-22 05:55:32,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-22 05:55:32,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-22 05:55:32,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-22 05:55:32,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-22 05:55:32,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-22 05:55:32,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-22 05:55:32,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-22 05:55:32,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-22 05:55:32,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-22 05:55:32,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-22 05:55:32,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-22 05:55:32,982 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-22 05:55:32,982 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1182921277:90 , srv1256262313:90 , srv1350285687:90 , srv138767701:90 , srv1428612270:90 , srv1461431:90 , srv152983832:90 , srv161798207:90 , srv1621437292:90 , srv1630429571:90 , srv163944081:90 , srv1958090542:90 , srv2071952832:90 , srv20902685:90 , srv2111316196:90 , srv2136561743:90 , srv2145918736:90 , srv279001139:90 , srv33233295:90 , srv388135465:90 , srv412402871:90 , srv635101089:90 , srv690448789:90 , srv71873275:90 , srv725032310:90 , srv756175702:90 , srv796216470:90 , srv797373847:90 , srv934883970:90 , srv970295007:90 } 2023-10-22 05:55:32,997 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=234 (was 405), ProcessCount=170 (was 172), AvailableMemoryMB=7992 (was 8274)