2023-08-12 11:53:11,125 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-12 11:53:11,446 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-12 11:53:11,560 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-12 11:53:11,632 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-12 11:53:11,641 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-12 11:53:11,656 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=314, ProcessCount=171, AvailableMemoryMB=23235 2023-08-12 11:53:11,659 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-12 11:53:11,659 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-12 11:53:11,812 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001998548:84 , srv1040010056:84 , srv1043113905:84 , srv1072518563:84 , srv1081568637:84 , srv1081677579:84 , srv1179943911:84 , srv120090697:84 , srv123621579:84 , srv1236280057:84 , srv1238346151:84 , srv1240696195:84 , srv1247360653:84 , srv1288574523:84 , srv1306875697:84 , srv1310709096:84 , srv1316446156:84 , srv1327129220:84 , srv1340447196:84 , srv1354643278:84 , srv1356580531:84 , srv138837792:84 , srv1390887940:84 , srv1399474316:84 , srv1417067690:84 , srv1417082268:84 , srv1467303206:84 , srv1480639335:84 , srv1495487808:84 , srv1519592121:84 , srv1597972963:84 , srv1602706886:84 , srv1616131353:84 , srv1631438074:84 , srv1645688024:84 , srv1656236604:84 , srv1682929361:84 , srv1695489959:84 , srv1696151510:84 , srv1699710598:84 , srv172525360:84 , srv1734572057:84 , srv180777126:84 , srv1825301387:84 , srv1836471882:84 , srv1867640675:84 , srv188171231:84 , srv1883979570:84 , srv1886831243:84 , srv1975743593:84 , srv1975837108:84 , srv1980219627:84 , srv1994322936:84 , srv2027543065:84 , srv2029381681:84 , srv2059627261:84 , srv2093040501:84 , srv2103685828:84 , srv2105611268:84 , srv2123704244:84 , srv2145732481:84 , srv239712892:84 , srv277201166:84 , srv320027657:84 , srv329340528:84 , srv362191261:84 , srv378470091:84 , srv396603580:84 , srv410709073:84 , srv434503807:84 , srv486257734:84 , srv536210809:84 , srv567614300:84 , srv572595575:84 , srv588266623:84 , srv606621377:84 , srv614863756:84 , srv635110307:84 , srv661070147:84 , srv697981035:84 , srv702606574:84 , srv708641490:84 , srv711240651:84 , srv716067116:84 , srv728794279:84 , srv758122008:84 , srv776762348:84 , srv808496645:84 , srv852770466:84 , srv863062294:84 , srv867762073:84 , srv885231926:84 , srv914441906:84 , srv93593817:84 , srv959650311:84 , srv964125847:84 , srv965383905:84 , srv992130151:84 , srv996456818:84 , srv1993002656:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-12 11:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-12 11:53:11,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980219627=51, srv1886831243=48, srv711240651=83, srv728794279=85, srv1825301387=43, srv885231926=92, srv1390887940=22, srv486257734=71, srv964125847=96, srv1696151510=38, srv2103685828=58, srv239712892=62, srv536210809=72, srv93593817=94, srv1040010056=1, srv1616131353=32, srv123621579=8, srv1399474316=23, srv2145732481=61, srv1240696195=11, srv434503807=70, srv1179943911=6, srv2105611268=59, srv2029381681=55, srv867762073=91, srv1519592121=29, srv758122008=86, srv1836471882=44, srv1734572057=41, srv329340528=65, srv1306875697=14, srv606621377=76, srv1480639335=27, srv1695489959=37, srv1356580531=20, srv1340447196=18, srv965383905=97, srv614863756=77, srv1327129220=17, srv808496645=88, srv1994322936=53, srv1238346151=10, srv572595575=74, srv1354643278=19, srv138837792=21, srv1631438074=33, srv702606574=81, srv1288574523=13, srv708641490=82, srv1645688024=34, srv567614300=73, srv1656236604=35, srv1993002656=52, srv188171231=46, srv2123704244=60, srv1081568637=4, srv172525360=40, srv180777126=42, srv2027543065=54, srv1975743593=49, srv320027657=64, srv992130151=98, srv852770466=89, srv1975837108=50, srv1495487808=28, srv776762348=87, srv959650311=95, srv635110307=78, srv2093040501=57, srv863062294=90, srv1316446156=16, srv1247360653=12, srv396603580=68, srv1883979570=47, srv1597972963=30, srv996456818=99, srv410709073=69, srv1467303206=26, srv588266623=75, srv1081677579=5, srv716067116=84, srv1699710598=39, srv277201166=63, srv1236280057=9, srv2059627261=56, srv1072518563=3, srv1417067690=24, srv1602706886=31, srv378470091=67, srv661070147=79, srv120090697=7, srv697981035=80, srv1001998548=0, srv1043113905=2, srv362191261=66, srv1310709096=15, srv1417082268=25, srv1867640675=45, srv914441906=93, srv1682929361=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-12 11:53:11,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-12 11:53:11,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-12 11:53:11,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-12 11:53:11,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-12 11:53:11,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-12 11:53:11,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-12 11:53:11,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-12 11:53:11,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-12 11:53:11,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-12 11:53:11,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-12 11:53:11,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-12 11:53:11,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-12 11:53:11,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-12 11:53:11,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-12 11:53:11,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-12 11:53:11,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-12 11:53:11,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-12 11:53:11,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-12 11:53:11,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-12 11:53:11,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-12 11:53:11,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-12 11:53:11,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-12 11:53:11,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-12 11:53:11,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-12 11:53:11,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-12 11:53:11,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-12 11:53:11,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-12 11:53:11,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-12 11:53:11,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-12 11:53:11,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-12 11:53:11,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-12 11:53:11,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-12 11:53:11,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-12 11:53:11,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-12 11:53:11,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-12 11:53:11,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-12 11:53:11,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-12 11:53:11,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-12 11:53:11,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-12 11:53:11,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-12 11:53:11,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-12 11:53:11,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-12 11:53:11,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-12 11:53:11,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-12 11:53:11,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-12 11:53:11,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-12 11:53:11,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-12 11:53:11,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-12 11:53:11,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-12 11:53:11,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-12 11:53:11,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-12 11:53:11,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-12 11:53:11,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-12 11:53:11,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-12 11:53:11,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-12 11:53:11,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-12 11:53:11,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-12 11:53:11,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-12 11:53:11,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-12 11:53:11,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-12 11:53:11,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-12 11:53:11,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-12 11:53:11,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-12 11:53:11,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-12 11:53:11,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-12 11:53:11,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-12 11:53:11,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-12 11:53:11,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-12 11:53:11,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-12 11:53:11,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-12 11:53:11,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-12 11:53:11,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-12 11:53:11,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-12 11:53:11,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-12 11:53:11,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-12 11:53:11,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-12 11:53:11,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-12 11:53:11,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-12 11:53:11,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-12 11:53:11,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-12 11:53:11,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-12 11:53:11,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-12 11:53:11,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-12 11:53:11,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-12 11:53:11,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-12 11:53:11,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-12 11:53:11,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-12 11:53:11,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-12 11:53:11,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-12 11:53:11,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-12 11:53:11,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-12 11:53:11,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-12 11:53:11,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-12 11:53:11,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-12 11:53:11,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-12 11:53:11,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-12 11:53:11,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-12 11:53:11,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-12 11:53:11,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-12 11:53:11,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-12 11:53:11,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-12 11:53:12,118 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:53:12,119 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-12 11:53:12,119 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-12 11:53:12,530 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 590 ms to try 5000 different iterations. Found a solution that moves 4741 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1747935125556695. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04133029205711441, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0561463535688622, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5267777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04133029205711441, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1695, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24116666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:53:12,557 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1602706886:80 , srv1631438074:81 , srv277201166:81 , srv567614300:81 , srv120090697:82 , srv1247360653:82 , srv1356580531:82 , srv1695489959:82 , srv2027543065:82 , srv239712892:82 , srv728794279:82 , srv1288574523:83 , srv1417082268:83 , srv1480639335:83 , srv1682929361:83 , srv1825301387:83 , srv1980219627:83 , srv434503807:83 , srv852770466:83 , srv867762073:83 , srv1240696195:84 , srv1399474316:84 , srv1616131353:84 , srv180777126:84 , srv1975743593:84 , srv2145732481:84 , srv410709073:84 , srv711240651:84 , srv776762348:84 , srv808496645:84 , srv863062294:84 , srv959650311:84 , srv1072518563:85 , srv1081677579:85 , srv1495487808:85 , srv1519592121:85 , srv1734572057:85 , srv188171231:85 , srv1975837108:85 , srv362191261:85 , srv396603580:85 , srv614863756:85 , srv885231926:85 , srv93593817:85 , srv1340447196:86 , srv1656236604:86 , srv1994322936:86 , srv2059627261:86 , srv320027657:86 , srv1040010056:87 , srv1081568637:87 , srv1179943911:87 , srv1316446156:87 , srv1597972963:87 , srv1645688024:87 , srv1696151510:87 , srv1836471882:87 , srv1886831243:87 , srv2029381681:87 , srv2123704244:87 , srv697981035:87 , srv716067116:87 , srv992130151:87 , srv996456818:87 , srv1043113905:88 , srv1354643278:88 , srv138837792:88 , srv1417067690:88 , srv172525360:88 , srv2093040501:88 , srv572595575:88 , srv702606574:88 , srv708641490:88 , srv914441906:88 , srv1001998548:89 , srv1236280057:89 , srv1238346151:89 , srv1699710598:89 , srv1883979570:89 , srv378470091:89 , srv486257734:89 , srv536210809:89 , srv606621377:89 , srv635110307:89 , srv661070147:89 , srv965383905:89 , srv123621579:90 , srv1310709096:90 , srv588266623:90 , srv964125847:90 , srv1306875697:91 , srv1467303206:91 , srv1867640675:91 , srv2103685828:91 , srv2105611268:91 , srv329340528:91 , srv1390887940:93 , srv758122008:93 , srv1327129220:96 , srv1993002656:457 } 2023-08-12 11:53:12,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-12 11:53:12,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980219627=51, srv1886831243=48, srv711240651=83, srv728794279=85, srv1825301387=43, srv885231926=92, srv1390887940=22, srv486257734=71, srv964125847=96, srv1696151510=38, srv2103685828=58, srv239712892=62, srv536210809=72, srv93593817=94, srv1040010056=1, srv1616131353=32, srv123621579=8, srv1399474316=23, srv2145732481=61, srv1240696195=11, srv434503807=70, srv1179943911=6, srv2105611268=59, srv2029381681=55, srv867762073=91, srv1519592121=29, srv758122008=86, srv1836471882=44, srv1734572057=41, srv329340528=65, srv1306875697=14, srv606621377=76, srv1480639335=27, srv1695489959=37, srv1356580531=20, srv1340447196=18, srv965383905=97, srv614863756=77, srv1327129220=17, srv808496645=88, srv1994322936=53, srv1238346151=10, srv572595575=74, srv1354643278=19, srv138837792=21, srv1631438074=33, srv702606574=81, srv1288574523=13, srv708641490=82, srv1645688024=34, srv567614300=73, srv1656236604=35, srv1993002656=52, srv188171231=46, srv2123704244=60, srv1081568637=4, srv172525360=40, srv180777126=42, srv2027543065=54, srv1975743593=49, srv320027657=64, srv992130151=98, srv852770466=89, srv1975837108=50, srv1495487808=28, srv776762348=87, srv959650311=95, srv635110307=78, srv2093040501=57, srv863062294=90, srv1316446156=16, srv1247360653=12, srv396603580=68, srv1883979570=47, srv1597972963=30, srv996456818=99, srv410709073=69, srv1467303206=26, srv588266623=75, srv1081677579=5, srv716067116=84, srv1699710598=39, srv277201166=63, srv1236280057=9, srv2059627261=56, srv1072518563=3, srv1417067690=24, srv1602706886=31, srv378470091=67, srv661070147=79, srv120090697=7, srv697981035=80, srv1001998548=0, srv1043113905=2, srv362191261=66, srv1310709096=15, srv1417082268=25, srv1867640675=45, srv914441906=93, srv1682929361=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-12 11:53:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-12 11:53:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-12 11:53:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-12 11:53:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-12 11:53:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-12 11:53:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-12 11:53:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-12 11:53:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-12 11:53:12,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-12 11:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-12 11:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-12 11:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-12 11:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-12 11:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-12 11:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-12 11:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-12 11:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-12 11:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-12 11:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-12 11:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-12 11:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-12 11:53:12,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-12 11:53:12,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-12 11:53:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-12 11:53:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-12 11:53:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-12 11:53:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-12 11:53:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-12 11:53:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-12 11:53:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-12 11:53:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-12 11:53:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-12 11:53:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-12 11:53:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-12 11:53:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-12 11:53:12,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-12 11:53:12,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-12 11:53:12,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-12 11:53:12,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-12 11:53:12,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-12 11:53:12,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-12 11:53:12,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-12 11:53:12,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-12 11:53:12,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-12 11:53:12,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-12 11:53:12,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-12 11:53:12,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-12 11:53:12,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-12 11:53:12,723 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04133029205711441, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0561463535688622, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04133029205711441, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1695, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24116666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:53:12,724 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-12 11:53:12,724 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1747603153641317, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04133029205711441, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0561463535688622, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04133029205711441, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1695, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24116666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-12 11:53:13,115 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 490 ms to try 5000 different iterations. Found a solution that moves 2616 regions; Going from a computed imbalance of 0.1747603153641317 to a new imbalance of 0.015162643969718725. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022855728761060144); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0315837120078931, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2906666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022855728761060144); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04708333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:53:13,128 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1327129220:78 , srv606621377:79 , srv758122008:79 , srv1867640675:80 , srv1390887940:81 , srv1467303206:81 , srv2103685828:81 , srv2105611268:81 , srv486257734:81 , srv1001998548:82 , srv1236280057:82 , srv1238346151:82 , srv1310709096:82 , srv1883979570:82 , srv329340528:82 , srv588266623:82 , srv965383905:82 , srv1043113905:83 , srv1179943911:83 , srv1306875697:83 , srv1699710598:83 , srv1975837108:83 , srv536210809:83 , srv572595575:83 , srv661070147:83 , srv702606574:83 , srv123621579:84 , srv1316446156:84 , srv1340447196:84 , srv138837792:84 , srv1696151510:84 , srv172525360:84 , srv1886831243:84 , srv2093040501:84 , srv396603580:84 , srv697981035:84 , srv716067116:84 , srv964125847:84 , srv992130151:84 , srv1072518563:85 , srv1081568637:85 , srv1354643278:85 , srv1597972963:85 , srv1734572057:85 , srv1836471882:85 , srv2059627261:85 , srv2123704244:85 , srv2145732481:85 , srv708641490:85 , srv1240696195:86 , srv1495487808:86 , srv1519592121:86 , srv1645688024:86 , srv1656236604:86 , srv1994322936:86 , srv635110307:86 , srv711240651:86 , srv808496645:86 , srv914441906:86 , srv996456818:86 , srv1081677579:87 , srv1682929361:87 , srv180777126:87 , srv188171231:87 , srv320027657:87 , srv93593817:87 , srv959650311:87 , srv1040010056:88 , srv1399474316:88 , srv1417067690:88 , srv1417082268:88 , srv1975743593:88 , srv2027543065:88 , srv2029381681:88 , srv378470091:88 , srv410709073:88 , srv614863756:88 , srv776762348:88 , srv867762073:88 , srv1247360653:89 , srv1288574523:89 , srv1356580531:89 , srv1480639335:89 , srv1631438074:89 , srv1825301387:89 , srv362191261:89 , srv863062294:89 , srv885231926:89 , srv120090697:90 , srv1616131353:90 , srv277201166:90 , srv1602706886:91 , srv1695489959:91 , srv567614300:91 , srv728794279:91 , srv1980219627:92 , srv239712892:92 , srv434503807:92 , srv852770466:92 , srv1993002656:520 } 2023-08-12 11:53:13,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-12 11:53:13,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980219627=51, srv1886831243=48, srv711240651=83, srv728794279=85, srv1825301387=43, srv885231926=92, srv1390887940=22, srv486257734=71, srv964125847=96, srv1696151510=38, srv2103685828=58, srv239712892=62, srv536210809=72, srv93593817=94, srv1040010056=1, srv1616131353=32, srv123621579=8, srv1399474316=23, srv2145732481=61, srv1240696195=11, srv434503807=70, srv1179943911=6, srv2105611268=59, srv2029381681=55, srv867762073=91, srv1519592121=29, srv758122008=86, srv1836471882=44, srv1734572057=41, srv329340528=65, srv1306875697=14, srv606621377=76, srv1480639335=27, srv1695489959=37, srv1356580531=20, srv1340447196=18, srv965383905=97, srv614863756=77, srv1327129220=17, srv808496645=88, srv1994322936=53, srv1238346151=10, srv572595575=74, srv1354643278=19, srv138837792=21, srv1631438074=33, srv702606574=81, srv1288574523=13, srv708641490=82, srv1645688024=34, srv567614300=73, srv1656236604=35, srv1993002656=52, srv188171231=46, srv2123704244=60, srv1081568637=4, srv172525360=40, srv180777126=42, srv2027543065=54, srv1975743593=49, srv320027657=64, srv992130151=98, srv852770466=89, srv1975837108=50, srv1495487808=28, srv776762348=87, srv959650311=95, srv635110307=78, srv2093040501=57, srv863062294=90, srv1316446156=16, srv1247360653=12, srv396603580=68, srv1883979570=47, srv1597972963=30, srv996456818=99, srv410709073=69, srv1467303206=26, srv588266623=75, srv1081677579=5, srv716067116=84, srv1699710598=39, srv277201166=63, srv1236280057=9, srv2059627261=56, srv1072518563=3, srv1417067690=24, srv1602706886=31, srv378470091=67, srv661070147=79, srv120090697=7, srv697981035=80, srv1001998548=0, srv1043113905=2, srv362191261=66, srv1310709096=15, srv1417082268=25, srv1867640675=45, srv914441906=93, srv1682929361=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-12 11:53:13,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-12 11:53:13,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-12 11:53:13,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-12 11:53:13,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-12 11:53:13,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-12 11:53:13,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-12 11:53:13,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-12 11:53:13,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-12 11:53:13,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-12 11:53:13,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-12 11:53:13,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-12 11:53:13,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-12 11:53:13,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-12 11:53:13,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-12 11:53:13,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-12 11:53:13,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-12 11:53:13,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-12 11:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-12 11:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-12 11:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-12 11:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-12 11:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-12 11:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-12 11:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-12 11:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-12 11:53:13,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-12 11:53:13,294 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022855728761060144); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0315837120078931, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022855728761060144); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04708333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:53:13,295 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-12 11:53:13,295 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015144326346208308, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022855728761060144); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0315837120078931, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022855728761060144); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04708333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-12 11:53:13,669 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 502 ms to try 5000 different iterations. Found a solution that moves 1043 regions; Going from a computed imbalance of 0.015144326346208308 to a new imbalance of 0.0010328202493145838. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006662924501529447); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009026709338484401); MoveCostFunction : (multiplier=7.0, imbalance=0.11588888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006662924501529447); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:53:13,673 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv378470091:80 , srv1417067690:81 , srv1040010056:82 , srv1980219627:82 , srv2029381681:82 , srv434503807:82 , srv635110307:82 , srv123621579:83 , srv1306875697:83 , srv1327129220:83 , srv1390887940:83 , srv362191261:83 , srv852770466:83 , srv885231926:83 , srv914441906:83 , srv964125847:83 , srv1480639335:84 , srv1616131353:84 , srv1695489959:84 , srv1975743593:84 , srv2123704244:84 , srv239712892:84 , srv329340528:84 , srv614863756:84 , srv728794279:84 , srv1081568637:85 , srv1081677579:85 , srv120090697:85 , srv1236280057:85 , srv1354643278:85 , srv1399474316:85 , srv1645688024:85 , srv1656236604:85 , srv1699710598:85 , srv1994322936:85 , srv2103685828:85 , srv320027657:85 , srv536210809:85 , srv588266623:85 , srv758122008:85 , srv863062294:85 , srv93593817:85 , srv996456818:85 , srv1247360653:86 , srv1288574523:86 , srv1310709096:86 , srv138837792:86 , srv1417082268:86 , srv1467303206:86 , srv1495487808:86 , srv1519592121:86 , srv1602706886:86 , srv1682929361:86 , srv172525360:86 , srv1825301387:86 , srv1867640675:86 , srv188171231:86 , srv1883979570:86 , srv1886831243:86 , srv2059627261:86 , srv2093040501:86 , srv2105611268:86 , srv277201166:86 , srv410709073:86 , srv567614300:86 , srv661070147:86 , srv702606574:86 , srv708641490:86 , srv776762348:86 , srv867762073:86 , srv959650311:86 , srv965383905:86 , srv992130151:86 , srv1001998548:87 , srv1043113905:87 , srv1238346151:87 , srv1240696195:87 , srv1340447196:87 , srv1356580531:87 , srv1597972963:87 , srv1631438074:87 , srv1696151510:87 , srv1734572057:87 , srv180777126:87 , srv1836471882:87 , srv486257734:87 , srv572595575:87 , srv697981035:87 , srv711240651:87 , srv716067116:87 , srv1072518563:88 , srv1316446156:88 , srv2027543065:88 , srv808496645:88 , srv1179943911:89 , srv1975837108:89 , srv2145732481:89 , srv396603580:89 , srv606621377:89 , srv1993002656:540 } 2023-08-12 11:53:13,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-12 11:53:13,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980219627=51, srv1886831243=48, srv711240651=83, srv728794279=85, srv1825301387=43, srv885231926=92, srv1390887940=22, srv486257734=71, srv964125847=96, srv1696151510=38, srv2103685828=58, srv239712892=62, srv536210809=72, srv93593817=94, srv1040010056=1, srv1616131353=32, srv123621579=8, srv1399474316=23, srv2145732481=61, srv1240696195=11, srv434503807=70, srv1179943911=6, srv2105611268=59, srv2029381681=55, srv867762073=91, srv1519592121=29, srv758122008=86, srv1836471882=44, srv1734572057=41, srv329340528=65, srv1306875697=14, srv606621377=76, srv1480639335=27, srv1695489959=37, srv1356580531=20, srv1340447196=18, srv965383905=97, srv614863756=77, srv1327129220=17, srv808496645=88, srv1994322936=53, srv1238346151=10, srv572595575=74, srv1354643278=19, srv138837792=21, srv1631438074=33, srv702606574=81, srv1288574523=13, srv708641490=82, srv1645688024=34, srv567614300=73, srv1656236604=35, srv1993002656=52, srv188171231=46, srv2123704244=60, srv1081568637=4, srv172525360=40, srv180777126=42, srv2027543065=54, srv1975743593=49, srv320027657=64, srv992130151=98, srv852770466=89, srv1975837108=50, srv1495487808=28, srv776762348=87, srv959650311=95, srv635110307=78, srv2093040501=57, srv863062294=90, srv1316446156=16, srv1247360653=12, srv396603580=68, srv1883979570=47, srv1597972963=30, srv996456818=99, srv410709073=69, srv1467303206=26, srv588266623=75, srv1081677579=5, srv716067116=84, srv1699710598=39, srv277201166=63, srv1236280057=9, srv2059627261=56, srv1072518563=3, srv1417067690=24, srv1602706886=31, srv378470091=67, srv661070147=79, srv120090697=7, srv697981035=80, srv1001998548=0, srv1043113905=2, srv362191261=66, srv1310709096=15, srv1417082268=25, srv1867640675=45, srv914441906=93, srv1682929361=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-12 11:53:13,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-12 11:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-12 11:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-12 11:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-12 11:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-12 11:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-12 11:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-12 11:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-12 11:53:13,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-12 11:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-12 11:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-12 11:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-12 11:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-12 11:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-12 11:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-12 11:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-12 11:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-12 11:53:13,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-12 11:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-12 11:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-12 11:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-12 11:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-12 11:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-12 11:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-12 11:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-12 11:53:13,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-12 11:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-12 11:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-12 11:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-12 11:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-12 11:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-12 11:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-12 11:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-12 11:53:13,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-12 11:53:13,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-12 11:53:13,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-12 11:53:13,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-12 11:53:13,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-12 11:53:13,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-12 11:53:13,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-12 11:53:13,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-12 11:53:13,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-12 11:53:13,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-12 11:53:13,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-12 11:53:13,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-12 11:53:13,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-12 11:53:13,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-12 11:53:13,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-12 11:53:13,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-12 11:53:13,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-12 11:53:13,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-12 11:53:13,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-12 11:53:13,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-12 11:53:13,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-12 11:53:13,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-12 11:53:13,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-12 11:53:13,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-12 11:53:13,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-12 11:53:13,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-12 11:53:13,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-12 11:53:13,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-12 11:53:13,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-12 11:53:13,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-12 11:53:13,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-12 11:53:13,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-12 11:53:13,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-12 11:53:13,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-12 11:53:13,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-12 11:53:13,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-12 11:53:13,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-12 11:53:13,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-12 11:53:13,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-12 11:53:13,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-12 11:53:13,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-12 11:53:13,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-12 11:53:13,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-12 11:53:13,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-12 11:53:13,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-12 11:53:13,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-12 11:53:13,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-12 11:53:13,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-12 11:53:13,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-12 11:53:13,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-12 11:53:13,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-12 11:53:13,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-12 11:53:13,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-12 11:53:13,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-12 11:53:13,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-12 11:53:13,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-12 11:53:13,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-12 11:53:13,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-12 11:53:13,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-12 11:53:13,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-12 11:53:13,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-12 11:53:13,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-12 11:53:13,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-12 11:53:13,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-12 11:53:13,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-12 11:53:13,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-12 11:53:13,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-12 11:53:13,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-12 11:53:13,884 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006662924501529447); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009026709338484401); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006662924501529447); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:53:13,885 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-12 11:53:13,885 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0010255170072192604, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006662924501529447); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009026709338484401); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006662924501529447); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-12 11:53:14,338 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 595 ms to try 5000 different iterations. Found a solution that moves 310 regions; Going from a computed imbalance of 0.0010255170072192604 to a new imbalance of 4.215124924638174E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.034444444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:53:14,339 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1179943911:83 , srv1316446156:83 , srv1354643278:83 , srv1356580531:83 , srv1597972963:83 , srv180777126:83 , srv1836471882:83 , srv320027657:83 , srv697981035:83 , srv708641490:83 , srv1001998548:84 , srv1043113905:84 , srv1072518563:84 , srv1238346151:84 , srv1288574523:84 , srv138837792:84 , srv1417067690:84 , srv1467303206:84 , srv1495487808:84 , srv1616131353:84 , srv1645688024:84 , srv1696151510:84 , srv172525360:84 , srv1825301387:84 , srv1867640675:84 , srv188171231:84 , srv2027543065:84 , srv2093040501:84 , srv2105611268:84 , srv239712892:84 , srv396603580:84 , srv410709073:84 , srv567614300:84 , srv572595575:84 , srv661070147:84 , srv716067116:84 , srv776762348:84 , srv808496645:84 , srv852770466:84 , srv885231926:84 , srv996456818:84 , srv1040010056:85 , srv1081568637:85 , srv1081677579:85 , srv120090697:85 , srv123621579:85 , srv1240696195:85 , srv1247360653:85 , srv1306875697:85 , srv1310709096:85 , srv1327129220:85 , srv1340447196:85 , srv1390887940:85 , srv1399474316:85 , srv1417082268:85 , srv1519592121:85 , srv1602706886:85 , srv1631438074:85 , srv1656236604:85 , srv1695489959:85 , srv1699710598:85 , srv1883979570:85 , srv1886831243:85 , srv1975837108:85 , srv1980219627:85 , srv2029381681:85 , srv2059627261:85 , srv2103685828:85 , srv2145732481:85 , srv362191261:85 , srv378470091:85 , srv434503807:85 , srv486257734:85 , srv536210809:85 , srv588266623:85 , srv606621377:85 , srv635110307:85 , srv702606574:85 , srv711240651:85 , srv728794279:85 , srv758122008:85 , srv863062294:85 , srv867762073:85 , srv914441906:85 , srv959650311:85 , srv964125847:85 , srv965383905:85 , srv992130151:85 , srv1236280057:86 , srv1480639335:86 , srv1682929361:86 , srv1734572057:86 , srv1975743593:86 , srv1994322936:86 , srv2123704244:86 , srv277201166:86 , srv329340528:86 , srv614863756:86 , srv93593817:86 , srv1993002656:625 } 2023-08-12 11:53:14,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-12 11:53:14,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980219627=51, srv1886831243=48, srv711240651=83, srv728794279=85, srv1825301387=43, srv885231926=92, srv1390887940=22, srv486257734=71, srv964125847=96, srv1696151510=38, srv2103685828=58, srv239712892=62, srv536210809=72, srv93593817=94, srv1040010056=1, srv1616131353=32, srv123621579=8, srv1399474316=23, srv2145732481=61, srv1240696195=11, srv434503807=70, srv1179943911=6, srv2105611268=59, srv2029381681=55, srv867762073=91, srv1519592121=29, srv758122008=86, srv1836471882=44, srv1734572057=41, srv329340528=65, srv1306875697=14, srv606621377=76, srv1480639335=27, srv1695489959=37, srv1356580531=20, srv1340447196=18, srv965383905=97, srv614863756=77, srv1327129220=17, srv808496645=88, srv1994322936=53, srv1238346151=10, srv572595575=74, srv1354643278=19, srv138837792=21, srv1631438074=33, srv702606574=81, srv1288574523=13, srv708641490=82, srv1645688024=34, srv567614300=73, srv1656236604=35, srv1993002656=52, srv188171231=46, srv2123704244=60, srv1081568637=4, srv172525360=40, srv180777126=42, srv2027543065=54, srv1975743593=49, srv320027657=64, srv992130151=98, srv852770466=89, srv1975837108=50, srv1495487808=28, srv776762348=87, srv959650311=95, srv635110307=78, srv2093040501=57, srv863062294=90, srv1316446156=16, srv1247360653=12, srv396603580=68, srv1883979570=47, srv1597972963=30, srv996456818=99, srv410709073=69, srv1467303206=26, srv588266623=75, srv1081677579=5, srv716067116=84, srv1699710598=39, srv277201166=63, srv1236280057=9, srv2059627261=56, srv1072518563=3, srv1417067690=24, srv1602706886=31, srv378470091=67, srv661070147=79, srv120090697=7, srv697981035=80, srv1001998548=0, srv1043113905=2, srv362191261=66, srv1310709096=15, srv1417082268=25, srv1867640675=45, srv914441906=93, srv1682929361=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-12 11:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-12 11:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-12 11:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-12 11:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-12 11:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-12 11:53:14,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-12 11:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-12 11:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-12 11:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-12 11:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-12 11:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-12 11:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-12 11:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-12 11:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-12 11:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-12 11:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-12 11:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-12 11:53:14,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-12 11:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-12 11:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-12 11:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-12 11:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-12 11:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-12 11:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-12 11:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-12 11:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-12 11:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-12 11:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-12 11:53:14,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-12 11:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-12 11:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-12 11:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-12 11:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-12 11:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-12 11:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-12 11:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-12 11:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-12 11:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-12 11:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-12 11:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-12 11:53:14,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-12 11:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-12 11:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-12 11:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-12 11:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-12 11:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-12 11:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-12 11:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-12 11:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-12 11:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-12 11:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-12 11:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-12 11:53:14,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-12 11:53:14,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-12 11:53:14,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-12 11:53:14,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-12 11:53:14,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-12 11:53:14,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-12 11:53:14,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-12 11:53:14,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-12 11:53:14,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-12 11:53:14,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-12 11:53:14,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-12 11:53:14,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-12 11:53:14,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-12 11:53:14,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-12 11:53:14,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-12 11:53:14,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-12 11:53:14,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-12 11:53:14,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-12 11:53:14,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-12 11:53:14,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-12 11:53:14,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-12 11:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-12 11:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-12 11:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-12 11:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-12 11:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-12 11:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-12 11:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-12 11:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-12 11:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-12 11:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-12 11:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-12 11:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-12 11:53:14,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-12 11:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-12 11:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-12 11:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-12 11:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-12 11:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-12 11:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-12 11:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-12 11:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-12 11:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-12 11:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-12 11:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-12 11:53:14,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-12 11:53:14,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-12 11:53:14,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-12 11:53:14,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-12 11:53:14,528 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:53:14,528 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-12 11:53:14,529 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.9980582851798604E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-12 11:53:14,892 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 505 ms to try 5000 different iterations. Found a solution that moves 29 regions; Going from a computed imbalance of 3.9980582851798604E-5 to a new imbalance of 5.845555333665096E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0012535023884839248); MoveCostFunction : (multiplier=7.0, imbalance=0.0032222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:53:14,893 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1310709096:83 , srv1734572057:83 , srv1994322936:83 , srv2145732481:83 , srv277201166:83 , srv329340528:83 , srv614863756:83 , srv863062294:83 , srv93593817:83 , srv1001998548:84 , srv1040010056:84 , srv1043113905:84 , srv1072518563:84 , srv1081568637:84 , srv1081677579:84 , srv1179943911:84 , srv120090697:84 , srv123621579:84 , srv1236280057:84 , srv1238346151:84 , srv1240696195:84 , srv1247360653:84 , srv1288574523:84 , srv1306875697:84 , srv1316446156:84 , srv1327129220:84 , srv1340447196:84 , srv138837792:84 , srv1390887940:84 , srv1399474316:84 , srv1417082268:84 , srv1467303206:84 , srv1480639335:84 , srv1519592121:84 , srv1597972963:84 , srv1602706886:84 , srv1616131353:84 , srv1631438074:84 , srv1645688024:84 , srv1656236604:84 , srv1682929361:84 , srv1695489959:84 , srv1696151510:84 , srv1699710598:84 , srv172525360:84 , srv1825301387:84 , srv1836471882:84 , srv188171231:84 , srv1883979570:84 , srv1886831243:84 , srv1975743593:84 , srv1975837108:84 , srv1980219627:84 , srv2027543065:84 , srv2029381681:84 , srv2059627261:84 , srv2093040501:84 , srv2103685828:84 , srv2105611268:84 , srv2123704244:84 , srv239712892:84 , srv362191261:84 , srv378470091:84 , srv396603580:84 , srv410709073:84 , srv434503807:84 , srv486257734:84 , srv536210809:84 , srv567614300:84 , srv572595575:84 , srv588266623:84 , srv606621377:84 , srv635110307:84 , srv661070147:84 , srv702606574:84 , srv708641490:84 , srv711240651:84 , srv716067116:84 , srv728794279:84 , srv758122008:84 , srv776762348:84 , srv808496645:84 , srv852770466:84 , srv867762073:84 , srv914441906:84 , srv959650311:84 , srv964125847:84 , srv965383905:84 , srv992130151:84 , srv996456818:84 , srv1354643278:85 , srv1356580531:85 , srv1417067690:85 , srv1495487808:85 , srv180777126:85 , srv1867640675:85 , srv320027657:85 , srv697981035:85 , srv885231926:85 , srv1993002656:684 } 2023-08-12 11:53:14,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-12 11:53:14,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980219627=51, srv1886831243=48, srv711240651=83, srv728794279=85, srv1825301387=43, srv885231926=92, srv1390887940=22, srv486257734=71, srv964125847=96, srv1696151510=38, srv2103685828=58, srv239712892=62, srv536210809=72, srv93593817=94, srv1040010056=1, srv1616131353=32, srv123621579=8, srv1399474316=23, srv2145732481=61, srv1240696195=11, srv434503807=70, srv1179943911=6, srv2105611268=59, srv2029381681=55, srv867762073=91, srv1519592121=29, srv758122008=86, srv1836471882=44, srv1734572057=41, srv329340528=65, srv1306875697=14, srv606621377=76, srv1480639335=27, srv1695489959=37, srv1356580531=20, srv1340447196=18, srv965383905=97, srv614863756=77, srv1327129220=17, srv808496645=88, srv1994322936=53, srv1238346151=10, srv572595575=74, srv1354643278=19, srv138837792=21, srv1631438074=33, srv702606574=81, srv1288574523=13, srv708641490=82, srv1645688024=34, srv567614300=73, srv1656236604=35, srv1993002656=52, srv188171231=46, srv2123704244=60, srv1081568637=4, srv172525360=40, srv180777126=42, srv2027543065=54, srv1975743593=49, srv320027657=64, srv992130151=98, srv852770466=89, srv1975837108=50, srv1495487808=28, srv776762348=87, srv959650311=95, srv635110307=78, srv2093040501=57, srv863062294=90, srv1316446156=16, srv1247360653=12, srv396603580=68, srv1883979570=47, srv1597972963=30, srv996456818=99, srv410709073=69, srv1467303206=26, srv588266623=75, srv1081677579=5, srv716067116=84, srv1699710598=39, srv277201166=63, srv1236280057=9, srv2059627261=56, srv1072518563=3, srv1417067690=24, srv1602706886=31, srv378470091=67, srv661070147=79, srv120090697=7, srv697981035=80, srv1001998548=0, srv1043113905=2, srv362191261=66, srv1310709096=15, srv1417082268=25, srv1867640675=45, srv914441906=93, srv1682929361=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-12 11:53:14,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-12 11:53:14,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-12 11:53:14,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-12 11:53:14,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-12 11:53:14,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-12 11:53:14,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-12 11:53:14,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-12 11:53:14,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-12 11:53:14,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-12 11:53:14,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-12 11:53:14,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-12 11:53:14,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-12 11:53:14,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-12 11:53:14,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-12 11:53:14,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-12 11:53:14,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-12 11:53:14,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-12 11:53:14,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-12 11:53:14,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-12 11:53:14,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-12 11:53:14,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-12 11:53:14,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-12 11:53:14,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-12 11:53:14,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-12 11:53:14,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-12 11:53:14,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-12 11:53:14,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-12 11:53:14,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-12 11:53:14,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-12 11:53:14,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-12 11:53:14,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-12 11:53:14,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-12 11:53:14,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-12 11:53:14,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-12 11:53:14,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-12 11:53:14,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-12 11:53:14,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-12 11:53:14,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-12 11:53:14,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-12 11:53:14,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-12 11:53:14,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-12 11:53:14,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-12 11:53:14,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-12 11:53:14,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-12 11:53:14,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-12 11:53:14,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-12 11:53:14,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-12 11:53:14,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-12 11:53:14,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-12 11:53:14,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-12 11:53:14,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-12 11:53:14,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-12 11:53:14,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-12 11:53:14,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-12 11:53:14,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-12 11:53:14,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-12 11:53:14,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-12 11:53:14,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-12 11:53:14,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-12 11:53:14,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-12 11:53:14,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-12 11:53:14,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-12 11:53:14,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-12 11:53:14,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-12 11:53:14,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-12 11:53:14,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-12 11:53:14,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-12 11:53:14,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-12 11:53:14,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-12 11:53:14,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-12 11:53:14,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-12 11:53:14,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-12 11:53:14,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-12 11:53:14,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-12 11:53:14,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-12 11:53:14,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-12 11:53:14,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-12 11:53:14,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-12 11:53:14,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-12 11:53:14,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-12 11:53:14,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-12 11:53:14,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-12 11:53:14,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-12 11:53:14,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-12 11:53:14,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-12 11:53:14,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-12 11:53:14,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-12 11:53:14,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-12 11:53:14,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-12 11:53:14,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-12 11:53:14,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-12 11:53:14,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-12 11:53:14,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-12 11:53:14,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-12 11:53:14,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-12 11:53:14,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-12 11:53:14,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-12 11:53:14,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-12 11:53:14,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-12 11:53:14,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-12 11:53:14,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-12 11:53:15,061 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=5.642492993526674E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0012535023884839248); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:53:15,062 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1310709096:83 , srv1734572057:83 , srv1994322936:83 , srv2145732481:83 , srv277201166:83 , srv329340528:83 , srv614863756:83 , srv863062294:83 , srv93593817:83 , srv1001998548:84 , srv1040010056:84 , srv1043113905:84 , srv1072518563:84 , srv1081568637:84 , srv1081677579:84 , srv1179943911:84 , srv120090697:84 , srv123621579:84 , srv1236280057:84 , srv1238346151:84 , srv1240696195:84 , srv1247360653:84 , srv1288574523:84 , srv1306875697:84 , srv1316446156:84 , srv1327129220:84 , srv1340447196:84 , srv138837792:84 , srv1390887940:84 , srv1399474316:84 , srv1417082268:84 , srv1467303206:84 , srv1480639335:84 , srv1519592121:84 , srv1597972963:84 , srv1602706886:84 , srv1616131353:84 , srv1631438074:84 , srv1645688024:84 , srv1656236604:84 , srv1682929361:84 , srv1695489959:84 , srv1696151510:84 , srv1699710598:84 , srv172525360:84 , srv1825301387:84 , srv1836471882:84 , srv188171231:84 , srv1883979570:84 , srv1886831243:84 , srv1975743593:84 , srv1975837108:84 , srv1980219627:84 , srv2027543065:84 , srv2029381681:84 , srv2059627261:84 , srv2093040501:84 , srv2103685828:84 , srv2105611268:84 , srv2123704244:84 , srv239712892:84 , srv362191261:84 , srv378470091:84 , srv396603580:84 , srv410709073:84 , srv434503807:84 , srv486257734:84 , srv536210809:84 , srv567614300:84 , srv572595575:84 , srv588266623:84 , srv606621377:84 , srv635110307:84 , srv661070147:84 , srv702606574:84 , srv708641490:84 , srv711240651:84 , srv716067116:84 , srv728794279:84 , srv758122008:84 , srv776762348:84 , srv808496645:84 , srv852770466:84 , srv867762073:84 , srv914441906:84 , srv959650311:84 , srv964125847:84 , srv965383905:84 , srv992130151:84 , srv996456818:84 , srv1354643278:85 , srv1356580531:85 , srv1417067690:85 , srv1495487808:85 , srv180777126:85 , srv1867640675:85 , srv320027657:85 , srv697981035:85 , srv885231926:85 , srv1993002656:684 } 2023-08-12 11:53:15,087 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=385 (was 314) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=22337 (was 23235) 2023-08-12 11:53:15,103 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=385, ProcessCount=171, AvailableMemoryMB=22336 2023-08-12 11:53:15,104 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-12 11:53:15,104 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-12 11:53:15,118 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013523393:84 , srv1164806457:84 , srv1170844239:84 , srv1277921991:84 , srv1375360866:84 , srv149480085:84 , srv156239716:84 , srv1572854515:84 , srv1597762952:84 , srv1677279162:84 , srv176348974:84 , srv193859471:84 , srv2047760139:84 , srv2063441396:84 , srv2065979474:84 , srv209871963:84 , srv241237278:84 , srv426599462:84 , srv437628988:84 , srv495124659:84 , srv655367922:84 , srv663586158:84 , srv666806253:84 , srv719969947:84 , srv74300486:84 , srv823422635:84 , srv870283788:84 , srv948205801:84 , srv968185297:84 , srv731893568:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-12 11:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-12 11:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv426599462=17, srv437628988=18, srv193859471=11, srv1013523393=0, srv731893568=24, srv1170844239=2, srv209871963=15, srv1277921991=3, srv74300486=25, srv2047760139=12, srv2065979474=14, srv655367922=20, srv1375360866=4, srv1597762952=8, srv1572854515=7, srv149480085=5, srv823422635=26, srv968185297=29, srv870283788=27, srv495124659=19, srv2063441396=13, srv948205801=28, srv719969947=23, srv241237278=16, srv1164806457=1, srv663586158=21, srv666806253=22, srv156239716=6, srv1677279162=9, srv176348974=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-12 11:53:15,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-12 11:53:15,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-12 11:53:15,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-12 11:53:15,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-12 11:53:15,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-12 11:53:15,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-12 11:53:15,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-12 11:53:15,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-12 11:53:15,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-12 11:53:15,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-12 11:53:15,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-12 11:53:15,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-12 11:53:15,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-12 11:53:15,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-12 11:53:15,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-12 11:53:15,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-12 11:53:15,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-12 11:53:15,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-12 11:53:15,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-12 11:53:15,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-12 11:53:15,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-12 11:53:15,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-12 11:53:15,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-12 11:53:15,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-12 11:53:15,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-12 11:53:15,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-12 11:53:15,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-12 11:53:15,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-12 11:53:15,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-12 11:53:15,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-12 11:53:15,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-12 11:53:15,161 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1260775114836254, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:53:15,162 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912590309386154, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1260775114836254, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-12 11:55:15,140 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5579664 different iterations. Found a solution that moves 1861 regions; Going from a computed imbalance of 0.9912590309386154 to a new imbalance of 4.3436668390529226E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6892592592592592, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:55:15,146 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1013523393:90 , srv1164806457:90 , srv1170844239:90 , srv1277921991:90 , srv1375360866:90 , srv149480085:90 , srv156239716:90 , srv1572854515:90 , srv1597762952:90 , srv1677279162:90 , srv176348974:90 , srv193859471:90 , srv2047760139:90 , srv2063441396:90 , srv2065979474:90 , srv209871963:90 , srv241237278:90 , srv426599462:90 , srv437628988:90 , srv495124659:90 , srv655367922:90 , srv663586158:90 , srv666806253:90 , srv719969947:90 , srv731893568:90 , srv74300486:90 , srv823422635:90 , srv870283788:90 , srv948205801:90 , srv968185297:90 } 2023-08-12 11:55:15,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-12 11:55:15,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv426599462=17, srv437628988=18, srv193859471=11, srv1013523393=0, srv731893568=24, srv1170844239=2, srv209871963=15, srv1277921991=3, srv74300486=25, srv2047760139=12, srv2065979474=14, srv655367922=20, srv1375360866=4, srv1597762952=8, srv1572854515=7, srv149480085=5, srv823422635=26, srv968185297=29, srv870283788=27, srv495124659=19, srv2063441396=13, srv948205801=28, srv719969947=23, srv241237278=16, srv1164806457=1, srv663586158=21, srv666806253=22, srv156239716=6, srv1677279162=9, srv176348974=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-12 11:55:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-12 11:55:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-12 11:55:15,166 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-12 11:55:15,166 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1013523393:90 , srv1164806457:90 , srv1170844239:90 , srv1277921991:90 , srv1375360866:90 , srv149480085:90 , srv156239716:90 , srv1572854515:90 , srv1597762952:90 , srv1677279162:90 , srv176348974:90 , srv193859471:90 , srv2047760139:90 , srv2063441396:90 , srv2065979474:90 , srv209871963:90 , srv241237278:90 , srv426599462:90 , srv437628988:90 , srv495124659:90 , srv655367922:90 , srv663586158:90 , srv666806253:90 , srv719969947:90 , srv731893568:90 , srv74300486:90 , srv823422635:90 , srv870283788:90 , srv948205801:90 , srv968185297:90 } 2023-08-12 11:55:15,182 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=224 (was 385), ProcessCount=169 (was 171), AvailableMemoryMB=22241 (was 22336)