2023-02-13 04:56:24,364 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-02-13 04:56:24,671 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 04:56:24,772 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-13 04:56:24,829 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-13 04:56:24,837 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 04:56:24,849 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=113, ProcessCount=168, AvailableMemoryMB=4314 2023-02-13 04:56:24,851 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-13 04:56:24,851 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 04:56:24,968 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1034419777:84 , srv1034584917:84 , srv1060240232:84 , srv1062482022:84 , srv1063876296:84 , srv109327436:84 , srv1105602976:84 , srv1112562914:84 , srv1131675633:84 , srv113501901:84 , srv115523308:84 , srv1155894616:84 , srv1167800434:84 , srv1214976448:84 , srv1235178468:84 , srv1249496667:84 , srv1269671562:84 , srv1274089990:84 , srv1297933518:84 , srv1337322967:84 , srv1338127215:84 , srv1352266503:84 , srv1357236276:84 , srv1383643065:84 , srv1385395754:84 , srv1392639643:84 , srv1394949972:84 , srv1399077208:84 , srv1418977320:84 , srv1471737504:84 , srv148377448:84 , srv149554185:84 , srv149670675:84 , srv1525768503:84 , srv1537063970:84 , srv1568659895:84 , srv1569200767:84 , srv1573247162:84 , srv1687684332:84 , srv1689929802:84 , srv1696479207:84 , srv180261525:84 , srv1840443335:84 , srv1853656562:84 , srv1863916018:84 , srv1896032274:84 , srv1936112048:84 , srv1946568417:84 , srv1970823182:84 , srv1971433061:84 , srv1974812249:84 , srv2000522879:84 , srv2048355730:84 , srv2059347368:84 , srv2094208693:84 , srv2132463056:84 , srv2142821566:84 , srv284795410:84 , srv284890562:84 , srv293794740:84 , srv300263843:84 , srv319083211:84 , srv327875756:84 , srv342500229:84 , srv351765135:84 , srv353033660:84 , srv359313438:84 , srv382003520:84 , srv442584766:84 , srv446411558:84 , srv482752444:84 , srv490628626:84 , srv500602786:84 , srv513589526:84 , srv527123467:84 , srv544125714:84 , srv551729460:84 , srv575559925:84 , srv623507564:84 , srv629131828:84 , srv631002814:84 , srv649391136:84 , srv71819291:84 , srv731788543:84 , srv736877695:84 , srv739333305:84 , srv751235786:84 , srv772689098:84 , srv784611235:84 , srv795397999:84 , srv805846441:84 , srv81048323:84 , srv820529876:84 , srv843572118:84 , srv914013661:84 , srv945957236:84 , srv975930375:84 , srv976425185:84 , srv984196765:84 , srv2068150235:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-02-13 04:56:24,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 04:56:24,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2000522879=51, srv513589526=74, srv1155894616=11, srv1863916018=44, srv1034584917=1, srv843572118=94, srv1214976448=13, srv1235178468=14, srv284795410=58, srv1687684332=38, srv1689929802=39, srv772689098=88, srv945957236=96, srv1034419777=0, srv1338127215=20, srv1840443335=42, srv984196765=99, srv623507564=79, srv293794740=60, srv551729460=77, srv544125714=76, srv1269671562=16, srv1062482022=3, srv500602786=73, srv382003520=68, srv1418977320=28, srv975930375=97, srv1569200767=36, srv1971433061=49, srv1525768503=33, srv575559925=78, srv629131828=80, srv149670675=32, srv795397999=90, srv1537063970=34, srv1249496667=15, srv1936112048=46, srv115523308=10, srv631002814=81, srv2048355730=52, srv359313438=67, srv1352266503=21, srv71819291=83, srv353033660=66, srv1392639643=25, srv1853656562=43, srv442584766=69, srv1471737504=29, srv1167800434=12, srv751235786=87, srv1060240232=2, srv1131675633=8, srv1297933518=18, srv1974812249=50, srv319083211=62, srv976425185=98, srv81048323=92, srv1112562914=7, srv1337322967=19, srv2094208693=55, srv1399077208=27, srv327875756=63, srv2132463056=56, srv351765135=65, srv914013661=95, srv1573247162=37, srv1896032274=45, srv731788543=84, srv284890562=59, srv736877695=85, srv1357236276=22, srv149554185=31, srv2068150235=54, srv109327436=5, srv527123467=75, srv113501901=9, srv2142821566=57, srv784611235=89, srv1274089990=17, srv805846441=91, srv490628626=72, srv482752444=71, srv820529876=93, srv180261525=41, srv1946568417=47, srv446411558=70, srv1383643065=23, srv300263843=61, srv1970823182=48, srv1696479207=40, srv1105602976=6, srv1568659895=35, srv739333305=86, srv2059347368=53, srv1063876296=4, srv148377448=30, srv649391136=82, srv1385395754=24, srv1394949972=26, srv342500229=64} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 04:56:25,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:56:25,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 04:56:25,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 04:56:25,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 04:56:25,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 04:56:25,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 04:56:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 04:56:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 04:56:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 04:56:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 04:56:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 04:56:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 04:56:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 04:56:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 04:56:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 04:56:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 04:56:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 04:56:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 04:56:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 04:56:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 04:56:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 04:56:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 04:56:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 04:56:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 04:56:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 04:56:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 04:56:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 04:56:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 04:56:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 04:56:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 04:56:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 04:56:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 04:56:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 04:56:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 04:56:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 04:56:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 04:56:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 04:56:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 04:56:25,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 04:56:25,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 04:56:25,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 04:56:25,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 04:56:25,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 04:56:25,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 04:56:25,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 04:56:25,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 04:56:25,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 04:56:25,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 04:56:25,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 04:56:25,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 04:56:25,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 04:56:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 04:56:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 04:56:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 04:56:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 04:56:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 04:56:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 04:56:25,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 04:56:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 04:56:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 04:56:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 04:56:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 04:56:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 04:56:25,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 04:56:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 04:56:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 04:56:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 04:56:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 04:56:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 04:56:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 04:56:25,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 04:56:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 04:56:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 04:56:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 04:56:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 04:56:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 04:56:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 04:56:25,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 04:56:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 04:56:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 04:56:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 04:56:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 04:56:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 04:56:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 04:56:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 04:56:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 04:56:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 04:56:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 04:56:25,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 04:56:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 04:56:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 04:56:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 04:56:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 04:56:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 04:56:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 04:56:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 04:56:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 04:56:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 04:56:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 04:56:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 04:56:25,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 04:56:25,252 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:56:25,252 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 04:56:25,252 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 04:56:25,547 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 440 ms to try 5000 different iterations. Found a solution that moves 4662 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18203535733197845. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040880995187887285, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05388810787729364, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.518, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040880995187887285, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17733333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:56:25,569 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1249496667:81 , srv784611235:81 , srv945957236:81 , srv115523308:82 , srv1392639643:82 , srv1568659895:82 , srv180261525:82 , srv1062482022:83 , srv1112562914:83 , srv1214976448:83 , srv1269671562:83 , srv1689929802:83 , srv1974812249:83 , srv327875756:83 , srv544125714:83 , srv631002814:83 , srv975930375:83 , srv1399077208:84 , srv1687684332:84 , srv1863916018:84 , srv1896032274:84 , srv1971433061:84 , srv284795410:84 , srv300263843:84 , srv353033660:84 , srv442584766:84 , srv820529876:84 , srv1060240232:85 , srv1274089990:85 , srv1357236276:85 , srv148377448:85 , srv149554185:85 , srv1696479207:85 , srv2059347368:85 , srv482752444:85 , srv490628626:85 , srv513589526:85 , srv527123467:85 , srv629131828:85 , srv795397999:85 , srv976425185:85 , srv1034584917:86 , srv1155894616:86 , srv1337322967:86 , srv1338127215:86 , srv1383643065:86 , srv1385395754:86 , srv1573247162:86 , srv1936112048:86 , srv1970823182:86 , srv319083211:86 , srv382003520:86 , srv446411558:86 , srv500602786:86 , srv649391136:86 , srv751235786:86 , srv914013661:86 , srv1063876296:87 , srv1105602976:87 , srv1840443335:87 , srv1946568417:87 , srv2000522879:87 , srv284890562:87 , srv293794740:87 , srv351765135:87 , srv359313438:87 , srv71819291:87 , srv739333305:87 , srv805846441:87 , srv1034419777:88 , srv109327436:88 , srv1131675633:88 , srv1352266503:88 , srv1394949972:88 , srv1418977320:88 , srv1525768503:88 , srv1569200767:88 , srv2094208693:88 , srv551729460:88 , srv623507564:88 , srv984196765:88 , srv113501901:89 , srv1235178468:89 , srv1471737504:89 , srv149670675:89 , srv2048355730:89 , srv2132463056:89 , srv1537063970:90 , srv2142821566:90 , srv731788543:90 , srv736877695:90 , srv1853656562:91 , srv342500229:92 , srv772689098:92 , srv81048323:92 , srv843572118:92 , srv1167800434:93 , srv1297933518:95 , srv575559925:99 , srv2068150235:453 } 2023-02-13 04:56:25,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 04:56:25,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2000522879=51, srv513589526=74, srv1155894616=11, srv1863916018=44, srv1034584917=1, srv843572118=94, srv1214976448=13, srv1235178468=14, srv284795410=58, srv1687684332=38, srv1689929802=39, srv772689098=88, srv945957236=96, srv1034419777=0, srv1338127215=20, srv1840443335=42, srv984196765=99, srv623507564=79, srv293794740=60, srv551729460=77, srv544125714=76, srv1269671562=16, srv1062482022=3, srv500602786=73, srv382003520=68, srv1418977320=28, srv975930375=97, srv1569200767=36, srv1971433061=49, srv1525768503=33, srv575559925=78, srv629131828=80, srv149670675=32, srv795397999=90, srv1537063970=34, srv1249496667=15, srv1936112048=46, srv115523308=10, srv631002814=81, srv2048355730=52, srv359313438=67, srv1352266503=21, srv71819291=83, srv353033660=66, srv1392639643=25, srv1853656562=43, srv442584766=69, srv1471737504=29, srv1167800434=12, srv751235786=87, srv1060240232=2, srv1131675633=8, srv1297933518=18, srv1974812249=50, srv319083211=62, srv976425185=98, srv81048323=92, srv1112562914=7, srv1337322967=19, srv2094208693=55, srv1399077208=27, srv327875756=63, srv2132463056=56, srv351765135=65, srv914013661=95, srv1573247162=37, srv1896032274=45, srv731788543=84, srv284890562=59, srv736877695=85, srv1357236276=22, srv149554185=31, srv2068150235=54, srv109327436=5, srv527123467=75, srv113501901=9, srv2142821566=57, srv784611235=89, srv1274089990=17, srv805846441=91, srv490628626=72, srv482752444=71, srv820529876=93, srv180261525=41, srv1946568417=47, srv446411558=70, srv1383643065=23, srv300263843=61, srv1970823182=48, srv1696479207=40, srv1105602976=6, srv1568659895=35, srv739333305=86, srv2059347368=53, srv1063876296=4, srv148377448=30, srv649391136=82, srv1385395754=24, srv1394949972=26, srv342500229=64} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 04:56:25,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:56:25,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 04:56:25,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 04:56:25,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 04:56:25,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 04:56:25,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 04:56:25,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 04:56:25,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 04:56:25,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 04:56:25,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 04:56:25,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 04:56:25,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 04:56:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 04:56:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 04:56:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 04:56:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 04:56:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 04:56:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 04:56:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 04:56:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 04:56:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 04:56:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 04:56:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 04:56:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 04:56:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 04:56:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 04:56:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 04:56:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 04:56:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 04:56:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 04:56:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 04:56:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 04:56:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 04:56:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 04:56:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 04:56:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 04:56:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 04:56:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 04:56:25,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 04:56:25,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 04:56:25,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 04:56:25,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 04:56:25,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 04:56:25,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 04:56:25,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 04:56:25,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 04:56:25,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 04:56:25,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 04:56:25,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 04:56:25,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 04:56:25,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 04:56:25,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 04:56:25,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 04:56:25,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 04:56:25,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 04:56:25,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 04:56:25,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 04:56:25,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 04:56:25,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 04:56:25,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 04:56:25,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 04:56:25,742 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040880995187887285, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05388810787729364, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040880995187887285, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17733333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:56:25,742 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 04:56:25,743 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1820027133102638, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040880995187887285, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05388810787729364, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040880995187887285, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17733333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24341666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 04:56:26,046 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 427 ms to try 5000 different iterations. Found a solution that moves 2611 regions; Going from a computed imbalance of 0.1820027133102638 to a new imbalance of 0.01694935309780481. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022722758000263653); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033658348140558916, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2901111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022722758000263653); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:56:26,056 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv575559925:75 , srv1297933518:76 , srv772689098:79 , srv1167800434:80 , srv1537063970:80 , srv81048323:80 , srv1235178468:81 , srv1471737504:81 , srv342500229:81 , srv843572118:81 , srv1394949972:82 , srv149670675:82 , srv1853656562:82 , srv2142821566:82 , srv551729460:82 , srv731788543:82 , srv736877695:82 , srv113501901:83 , srv1352266503:83 , srv1569200767:83 , srv2048355730:83 , srv751235786:83 , srv109327436:84 , srv1131675633:84 , srv1418977320:84 , srv1525768503:84 , srv1946568417:84 , srv1970823182:84 , srv2132463056:84 , srv284890562:84 , srv739333305:84 , srv1034419777:85 , srv1105602976:85 , srv1338127215:85 , srv1357236276:85 , srv1696479207:85 , srv1840443335:85 , srv1936112048:85 , srv2094208693:85 , srv319083211:85 , srv351765135:85 , srv359313438:85 , srv446411558:85 , srv649391136:85 , srv71819291:85 , srv984196765:85 , srv1034584917:86 , srv1155894616:86 , srv148377448:86 , srv149554185:86 , srv1863916018:86 , srv1896032274:86 , srv2000522879:86 , srv293794740:86 , srv382003520:86 , srv482752444:86 , srv490628626:86 , srv623507564:86 , srv914013661:86 , srv1062482022:87 , srv1063876296:87 , srv1337322967:87 , srv1383643065:87 , srv1399077208:87 , srv1971433061:87 , srv353033660:87 , srv500602786:87 , srv527123467:87 , srv629131828:87 , srv976425185:87 , srv1060240232:88 , srv1112562914:88 , srv1214976448:88 , srv1269671562:88 , srv1568659895:88 , srv1573247162:88 , srv2059347368:88 , srv284795410:88 , srv544125714:88 , srv795397999:88 , srv805846441:88 , srv820529876:88 , srv1274089990:89 , srv1385395754:89 , srv1392639643:89 , srv1687684332:89 , srv1689929802:89 , srv1974812249:89 , srv327875756:89 , srv513589526:89 , srv1249496667:90 , srv631002814:90 , srv975930375:90 , srv180261525:91 , srv300263843:91 , srv784611235:91 , srv945957236:91 , srv115523308:93 , srv442584766:93 , srv2068150235:523 } 2023-02-13 04:56:26,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 04:56:26,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2000522879=51, srv513589526=74, srv1155894616=11, srv1863916018=44, srv1034584917=1, srv843572118=94, srv1214976448=13, srv1235178468=14, srv284795410=58, srv1687684332=38, srv1689929802=39, srv772689098=88, srv945957236=96, srv1034419777=0, srv1338127215=20, srv1840443335=42, srv984196765=99, srv623507564=79, srv293794740=60, srv551729460=77, srv544125714=76, srv1269671562=16, srv1062482022=3, srv500602786=73, srv382003520=68, srv1418977320=28, srv975930375=97, srv1569200767=36, srv1971433061=49, srv1525768503=33, srv575559925=78, srv629131828=80, srv149670675=32, srv795397999=90, srv1537063970=34, srv1249496667=15, srv1936112048=46, srv115523308=10, srv631002814=81, srv2048355730=52, srv359313438=67, srv1352266503=21, srv71819291=83, srv353033660=66, srv1392639643=25, srv1853656562=43, srv442584766=69, srv1471737504=29, srv1167800434=12, srv751235786=87, srv1060240232=2, srv1131675633=8, srv1297933518=18, srv1974812249=50, srv319083211=62, srv976425185=98, srv81048323=92, srv1112562914=7, srv1337322967=19, srv2094208693=55, srv1399077208=27, srv327875756=63, srv2132463056=56, srv351765135=65, srv914013661=95, srv1573247162=37, srv1896032274=45, srv731788543=84, srv284890562=59, srv736877695=85, srv1357236276=22, srv149554185=31, srv2068150235=54, srv109327436=5, srv527123467=75, srv113501901=9, srv2142821566=57, srv784611235=89, srv1274089990=17, srv805846441=91, srv490628626=72, srv482752444=71, srv820529876=93, srv180261525=41, srv1946568417=47, srv446411558=70, srv1383643065=23, srv300263843=61, srv1970823182=48, srv1696479207=40, srv1105602976=6, srv1568659895=35, srv739333305=86, srv2059347368=53, srv1063876296=4, srv148377448=30, srv649391136=82, srv1385395754=24, srv1394949972=26, srv342500229=64} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 04:56:26,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:56:26,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 04:56:26,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 04:56:26,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 04:56:26,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 04:56:26,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 04:56:26,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 04:56:26,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 04:56:26,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 04:56:26,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 04:56:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 04:56:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 04:56:26,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 04:56:26,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 04:56:26,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 04:56:26,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 04:56:26,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 04:56:26,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 04:56:26,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 04:56:26,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 04:56:26,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 04:56:26,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 04:56:26,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 04:56:26,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 04:56:26,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 04:56:26,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 04:56:26,190 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022722758000263653); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033658348140558916, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022722758000263653); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:56:26,190 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 04:56:26,190 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016931070485042693, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022722758000263653); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033658348140558916, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022722758000263653); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 04:56:26,453 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 360 ms to try 5000 different iterations. Found a solution that moves 994 regions; Going from a computed imbalance of 0.016931070485042693 to a new imbalance of 8.545398093993E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005325190191896699); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007595261089740389); MoveCostFunction : (multiplier=7.0, imbalance=0.11044444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005325190191896699); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:56:26,455 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv442584766:80 , srv1573247162:82 , srv300263843:82 , srv805846441:82 , srv1063876296:83 , srv115523308:83 , srv1274089990:83 , srv513589526:83 , srv623507564:83 , srv1337322967:84 , srv1383643065:84 , srv1385395754:84 , srv1853656562:84 , srv293794740:84 , srv500602786:84 , srv575559925:84 , srv631002814:84 , srv843572118:84 , srv1034419777:85 , srv1060240232:85 , srv109327436:85 , srv1105602976:85 , srv113501901:85 , srv1167800434:85 , srv1418977320:85 , srv1525768503:85 , srv1687684332:85 , srv1840443335:85 , srv1974812249:85 , srv2000522879:85 , srv2048355730:85 , srv2059347368:85 , srv2132463056:85 , srv284795410:85 , srv327875756:85 , srv359313438:85 , srv527123467:85 , srv71819291:85 , srv736877695:85 , srv784611235:85 , srv795397999:85 , srv914013661:85 , srv945957236:85 , srv976425185:85 , srv984196765:85 , srv1034584917:86 , srv1131675633:86 , srv1155894616:86 , srv1214976448:86 , srv1352266503:86 , srv1392639643:86 , srv1399077208:86 , srv148377448:86 , srv149670675:86 , srv1537063970:86 , srv1569200767:86 , srv1689929802:86 , srv180261525:86 , srv1936112048:86 , srv1971433061:86 , srv2094208693:86 , srv2142821566:86 , srv342500229:86 , srv351765135:86 , srv353033660:86 , srv382003520:86 , srv544125714:86 , srv629131828:86 , srv731788543:86 , srv739333305:86 , srv81048323:86 , srv820529876:86 , srv975930375:86 , srv1062482022:87 , srv1112562914:87 , srv1249496667:87 , srv1269671562:87 , srv1297933518:87 , srv1471737504:87 , srv149554185:87 , srv1896032274:87 , srv1946568417:87 , srv1970823182:87 , srv319083211:87 , srv446411558:87 , srv482752444:87 , srv490628626:87 , srv551729460:87 , srv649391136:87 , srv772689098:87 , srv1235178468:88 , srv1338127215:88 , srv1357236276:88 , srv1394949972:88 , srv1568659895:88 , srv1696479207:88 , srv1863916018:88 , srv284890562:88 , srv751235786:88 , srv2068150235:529 } 2023-02-13 04:56:26,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 04:56:26,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2000522879=51, srv513589526=74, srv1155894616=11, srv1863916018=44, srv1034584917=1, srv843572118=94, srv1214976448=13, srv1235178468=14, srv284795410=58, srv1687684332=38, srv1689929802=39, srv772689098=88, srv945957236=96, srv1034419777=0, srv1338127215=20, srv1840443335=42, srv984196765=99, srv623507564=79, srv293794740=60, srv551729460=77, srv544125714=76, srv1269671562=16, srv1062482022=3, srv500602786=73, srv382003520=68, srv1418977320=28, srv975930375=97, srv1569200767=36, srv1971433061=49, srv1525768503=33, srv575559925=78, srv629131828=80, srv149670675=32, srv795397999=90, srv1537063970=34, srv1249496667=15, srv1936112048=46, srv115523308=10, srv631002814=81, srv2048355730=52, srv359313438=67, srv1352266503=21, srv71819291=83, srv353033660=66, srv1392639643=25, srv1853656562=43, srv442584766=69, srv1471737504=29, srv1167800434=12, srv751235786=87, srv1060240232=2, srv1131675633=8, srv1297933518=18, srv1974812249=50, srv319083211=62, srv976425185=98, srv81048323=92, srv1112562914=7, srv1337322967=19, srv2094208693=55, srv1399077208=27, srv327875756=63, srv2132463056=56, srv351765135=65, srv914013661=95, srv1573247162=37, srv1896032274=45, srv731788543=84, srv284890562=59, srv736877695=85, srv1357236276=22, srv149554185=31, srv2068150235=54, srv109327436=5, srv527123467=75, srv113501901=9, srv2142821566=57, srv784611235=89, srv1274089990=17, srv805846441=91, srv490628626=72, srv482752444=71, srv820529876=93, srv180261525=41, srv1946568417=47, srv446411558=70, srv1383643065=23, srv300263843=61, srv1970823182=48, srv1696479207=40, srv1105602976=6, srv1568659895=35, srv739333305=86, srv2059347368=53, srv1063876296=4, srv148377448=30, srv649391136=82, srv1385395754=24, srv1394949972=26, srv342500229=64} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 04:56:26,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:56:26,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 04:56:26,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 04:56:26,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 04:56:26,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 04:56:26,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 04:56:26,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 04:56:26,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 04:56:26,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 04:56:26,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 04:56:26,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 04:56:26,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 04:56:26,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 04:56:26,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 04:56:26,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 04:56:26,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 04:56:26,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 04:56:26,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 04:56:26,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 04:56:26,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 04:56:26,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 04:56:26,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 04:56:26,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 04:56:26,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 04:56:26,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 04:56:26,609 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005325190191896699); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007595261089740389); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005325190191896699); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:56:26,609 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 04:56:26,610 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.47579672637314E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005325190191896699); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007595261089740389); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005325190191896699); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 04:56:26,908 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 412 ms to try 5000 different iterations. Found a solution that moves 241 regions; Going from a computed imbalance of 8.47579672637314E-4 to a new imbalance of 4.137728479609503E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.02677777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:56:26,909 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1338127215:83 , srv1385395754:83 , srv180261525:83 , srv2094208693:83 , srv284890562:83 , srv319083211:83 , srv975930375:83 , srv1034419777:84 , srv1034584917:84 , srv1060240232:84 , srv1112562914:84 , srv1131675633:84 , srv115523308:84 , srv1155894616:84 , srv1167800434:84 , srv1235178468:84 , srv1249496667:84 , srv1269671562:84 , srv1297933518:84 , srv1357236276:84 , srv1394949972:84 , srv149554185:84 , srv1568659895:84 , srv1687684332:84 , srv1689929802:84 , srv1840443335:84 , srv1863916018:84 , srv1946568417:84 , srv2000522879:84 , srv2059347368:84 , srv2132463056:84 , srv2142821566:84 , srv342500229:84 , srv351765135:84 , srv353033660:84 , srv382003520:84 , srv446411558:84 , srv500602786:84 , srv575559925:84 , srv629131828:84 , srv649391136:84 , srv731788543:84 , srv772689098:84 , srv784611235:84 , srv795397999:84 , srv81048323:84 , srv820529876:84 , srv984196765:84 , srv1062482022:85 , srv1063876296:85 , srv109327436:85 , srv1105602976:85 , srv113501901:85 , srv1214976448:85 , srv1274089990:85 , srv1337322967:85 , srv1352266503:85 , srv1383643065:85 , srv1392639643:85 , srv1399077208:85 , srv1418977320:85 , srv1471737504:85 , srv148377448:85 , srv149670675:85 , srv1525768503:85 , srv1569200767:85 , srv1696479207:85 , srv1853656562:85 , srv1896032274:85 , srv1936112048:85 , srv1970823182:85 , srv1971433061:85 , srv1974812249:85 , srv2048355730:85 , srv284795410:85 , srv293794740:85 , srv300263843:85 , srv327875756:85 , srv359313438:85 , srv442584766:85 , srv482752444:85 , srv490628626:85 , srv513589526:85 , srv527123467:85 , srv544125714:85 , srv551729460:85 , srv623507564:85 , srv631002814:85 , srv71819291:85 , srv736877695:85 , srv739333305:85 , srv751235786:85 , srv805846441:85 , srv843572118:85 , srv914013661:85 , srv976425185:85 , srv1537063970:86 , srv1573247162:86 , srv945957236:86 , srv2068150235:637 } 2023-02-13 04:56:26,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 04:56:26,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2000522879=51, srv513589526=74, srv1155894616=11, srv1863916018=44, srv1034584917=1, srv843572118=94, srv1214976448=13, srv1235178468=14, srv284795410=58, srv1687684332=38, srv1689929802=39, srv772689098=88, srv945957236=96, srv1034419777=0, srv1338127215=20, srv1840443335=42, srv984196765=99, srv623507564=79, srv293794740=60, srv551729460=77, srv544125714=76, srv1269671562=16, srv1062482022=3, srv500602786=73, srv382003520=68, srv1418977320=28, srv975930375=97, srv1569200767=36, srv1971433061=49, srv1525768503=33, srv575559925=78, srv629131828=80, srv149670675=32, srv795397999=90, srv1537063970=34, srv1249496667=15, srv1936112048=46, srv115523308=10, srv631002814=81, srv2048355730=52, srv359313438=67, srv1352266503=21, srv71819291=83, srv353033660=66, srv1392639643=25, srv1853656562=43, srv442584766=69, srv1471737504=29, srv1167800434=12, srv751235786=87, srv1060240232=2, srv1131675633=8, srv1297933518=18, srv1974812249=50, srv319083211=62, srv976425185=98, srv81048323=92, srv1112562914=7, srv1337322967=19, srv2094208693=55, srv1399077208=27, srv327875756=63, srv2132463056=56, srv351765135=65, srv914013661=95, srv1573247162=37, srv1896032274=45, srv731788543=84, srv284890562=59, srv736877695=85, srv1357236276=22, srv149554185=31, srv2068150235=54, srv109327436=5, srv527123467=75, srv113501901=9, srv2142821566=57, srv784611235=89, srv1274089990=17, srv805846441=91, srv490628626=72, srv482752444=71, srv820529876=93, srv180261525=41, srv1946568417=47, srv446411558=70, srv1383643065=23, srv300263843=61, srv1970823182=48, srv1696479207=40, srv1105602976=6, srv1568659895=35, srv739333305=86, srv2059347368=53, srv1063876296=4, srv148377448=30, srv649391136=82, srv1385395754=24, srv1394949972=26, srv342500229=64} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 04:56:26,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:56:26,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 04:56:26,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 04:56:26,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 04:56:26,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 04:56:26,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 04:56:26,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 04:56:26,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 04:56:26,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 04:56:26,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 04:56:26,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 04:56:26,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 04:56:26,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 04:56:26,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 04:56:26,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 04:56:26,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 04:56:26,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 04:56:26,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 04:56:26,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 04:56:26,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 04:56:26,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 04:56:26,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 04:56:26,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 04:56:26,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 04:56:26,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 04:56:26,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 04:56:26,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 04:56:26,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 04:56:26,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 04:56:26,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 04:56:26,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 04:56:26,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 04:56:27,033 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:56:27,033 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 04:56:27,034 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.9689766728048136E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 04:56:27,254 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 314 ms to try 5000 different iterations. Found a solution that moves 21 regions; Going from a computed imbalance of 3.9689766728048136E-5 to a new imbalance of 6.8911141966908056E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0023333333333333335); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:56:27,254 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1696479207:83 , srv342500229:83 , srv482752444:83 , srv490628626:83 , srv945957236:83 , srv1034419777:84 , srv1034584917:84 , srv1060240232:84 , srv1062482022:84 , srv1063876296:84 , srv109327436:84 , srv1105602976:84 , srv1112562914:84 , srv1131675633:84 , srv113501901:84 , srv115523308:84 , srv1155894616:84 , srv1167800434:84 , srv1214976448:84 , srv1235178468:84 , srv1249496667:84 , srv1269671562:84 , srv1274089990:84 , srv1297933518:84 , srv1337322967:84 , srv1338127215:84 , srv1352266503:84 , srv1357236276:84 , srv1383643065:84 , srv1385395754:84 , srv1392639643:84 , srv1394949972:84 , srv1399077208:84 , srv1418977320:84 , srv1471737504:84 , srv148377448:84 , srv149554185:84 , srv149670675:84 , srv1525768503:84 , srv1537063970:84 , srv1568659895:84 , srv1569200767:84 , srv1573247162:84 , srv1687684332:84 , srv1689929802:84 , srv180261525:84 , srv1853656562:84 , srv1863916018:84 , srv1896032274:84 , srv1936112048:84 , srv1946568417:84 , srv1970823182:84 , srv1971433061:84 , srv1974812249:84 , srv2000522879:84 , srv2048355730:84 , srv2059347368:84 , srv2094208693:84 , srv2132463056:84 , srv2142821566:84 , srv284795410:84 , srv284890562:84 , srv293794740:84 , srv300263843:84 , srv327875756:84 , srv351765135:84 , srv359313438:84 , srv382003520:84 , srv442584766:84 , srv446411558:84 , srv513589526:84 , srv527123467:84 , srv544125714:84 , srv551729460:84 , srv575559925:84 , srv623507564:84 , srv629131828:84 , srv631002814:84 , srv649391136:84 , srv71819291:84 , srv731788543:84 , srv736877695:84 , srv739333305:84 , srv751235786:84 , srv772689098:84 , srv795397999:84 , srv805846441:84 , srv81048323:84 , srv820529876:84 , srv843572118:84 , srv914013661:84 , srv975930375:84 , srv976425185:84 , srv984196765:84 , srv1840443335:85 , srv319083211:85 , srv353033660:85 , srv500602786:85 , srv784611235:85 , srv2068150235:684 } 2023-02-13 04:56:27,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 04:56:27,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2000522879=51, srv513589526=74, srv1155894616=11, srv1863916018=44, srv1034584917=1, srv843572118=94, srv1214976448=13, srv1235178468=14, srv284795410=58, srv1687684332=38, srv1689929802=39, srv772689098=88, srv945957236=96, srv1034419777=0, srv1338127215=20, srv1840443335=42, srv984196765=99, srv623507564=79, srv293794740=60, srv551729460=77, srv544125714=76, srv1269671562=16, srv1062482022=3, srv500602786=73, srv382003520=68, srv1418977320=28, srv975930375=97, srv1569200767=36, srv1971433061=49, srv1525768503=33, srv575559925=78, srv629131828=80, srv149670675=32, srv795397999=90, srv1537063970=34, srv1249496667=15, srv1936112048=46, srv115523308=10, srv631002814=81, srv2048355730=52, srv359313438=67, srv1352266503=21, srv71819291=83, srv353033660=66, srv1392639643=25, srv1853656562=43, srv442584766=69, srv1471737504=29, srv1167800434=12, srv751235786=87, srv1060240232=2, srv1131675633=8, srv1297933518=18, srv1974812249=50, srv319083211=62, srv976425185=98, srv81048323=92, srv1112562914=7, srv1337322967=19, srv2094208693=55, srv1399077208=27, srv327875756=63, srv2132463056=56, srv351765135=65, srv914013661=95, srv1573247162=37, srv1896032274=45, srv731788543=84, srv284890562=59, srv736877695=85, srv1357236276=22, srv149554185=31, srv2068150235=54, srv109327436=5, srv527123467=75, srv113501901=9, srv2142821566=57, srv784611235=89, srv1274089990=17, srv805846441=91, srv490628626=72, srv482752444=71, srv820529876=93, srv180261525=41, srv1946568417=47, srv446411558=70, srv1383643065=23, srv300263843=61, srv1970823182=48, srv1696479207=40, srv1105602976=6, srv1568659895=35, srv739333305=86, srv2059347368=53, srv1063876296=4, srv148377448=30, srv649391136=82, srv1385395754=24, srv1394949972=26, srv342500229=64} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 04:56:27,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:56:27,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 04:56:27,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 04:56:27,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 04:56:27,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 04:56:27,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 04:56:27,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 04:56:27,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 04:56:27,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 04:56:27,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 04:56:27,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 04:56:27,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 04:56:27,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 04:56:27,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 04:56:27,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 04:56:27,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 04:56:27,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 04:56:27,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 04:56:27,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 04:56:27,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 04:56:27,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 04:56:27,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 04:56:27,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 04:56:27,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 04:56:27,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 04:56:27,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 04:56:27,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 04:56:27,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 04:56:27,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 04:56:27,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 04:56:27,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 04:56:27,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 04:56:27,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 04:56:27,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 04:56:27,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 04:56:27,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 04:56:27,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 04:56:27,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 04:56:27,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 04:56:27,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 04:56:27,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 04:56:27,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 04:56:27,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 04:56:27,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 04:56:27,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 04:56:27,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 04:56:27,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 04:56:27,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 04:56:27,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 04:56:27,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 04:56:27,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 04:56:27,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 04:56:27,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 04:56:27,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 04:56:27,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 04:56:27,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 04:56:27,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 04:56:27,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 04:56:27,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 04:56:27,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 04:56:27,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 04:56:27,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 04:56:27,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 04:56:27,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 04:56:27,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 04:56:27,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 04:56:27,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 04:56:27,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 04:56:27,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 04:56:27,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 04:56:27,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 04:56:27,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 04:56:27,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 04:56:27,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 04:56:27,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 04:56:27,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 04:56:27,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 04:56:27,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 04:56:27,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 04:56:27,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 04:56:27,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 04:56:27,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 04:56:27,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 04:56:27,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 04:56:27,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 04:56:27,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 04:56:27,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 04:56:27,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 04:56:27,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 04:56:27,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 04:56:27,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 04:56:27,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 04:56:27,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 04:56:27,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 04:56:27,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 04:56:27,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 04:56:27,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 04:56:27,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 04:56:27,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 04:56:27,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 04:56:27,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 04:56:27,389 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:56:27,389 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1696479207:83 , srv342500229:83 , srv482752444:83 , srv490628626:83 , srv945957236:83 , srv1034419777:84 , srv1034584917:84 , srv1060240232:84 , srv1062482022:84 , srv1063876296:84 , srv109327436:84 , srv1105602976:84 , srv1112562914:84 , srv1131675633:84 , srv113501901:84 , srv115523308:84 , srv1155894616:84 , srv1167800434:84 , srv1214976448:84 , srv1235178468:84 , srv1249496667:84 , srv1269671562:84 , srv1274089990:84 , srv1297933518:84 , srv1337322967:84 , srv1338127215:84 , srv1352266503:84 , srv1357236276:84 , srv1383643065:84 , srv1385395754:84 , srv1392639643:84 , srv1394949972:84 , srv1399077208:84 , srv1418977320:84 , srv1471737504:84 , srv148377448:84 , srv149554185:84 , srv149670675:84 , srv1525768503:84 , srv1537063970:84 , srv1568659895:84 , srv1569200767:84 , srv1573247162:84 , srv1687684332:84 , srv1689929802:84 , srv180261525:84 , srv1853656562:84 , srv1863916018:84 , srv1896032274:84 , srv1936112048:84 , srv1946568417:84 , srv1970823182:84 , srv1971433061:84 , srv1974812249:84 , srv2000522879:84 , srv2048355730:84 , srv2059347368:84 , srv2094208693:84 , srv2132463056:84 , srv2142821566:84 , srv284795410:84 , srv284890562:84 , srv293794740:84 , srv300263843:84 , srv327875756:84 , srv351765135:84 , srv359313438:84 , srv382003520:84 , srv442584766:84 , srv446411558:84 , srv513589526:84 , srv527123467:84 , srv544125714:84 , srv551729460:84 , srv575559925:84 , srv623507564:84 , srv629131828:84 , srv631002814:84 , srv649391136:84 , srv71819291:84 , srv731788543:84 , srv736877695:84 , srv739333305:84 , srv751235786:84 , srv772689098:84 , srv795397999:84 , srv805846441:84 , srv81048323:84 , srv820529876:84 , srv843572118:84 , srv914013661:84 , srv975930375:84 , srv976425185:84 , srv984196765:84 , srv1840443335:85 , srv319083211:85 , srv353033660:85 , srv500602786:85 , srv784611235:85 , srv2068150235:684 } 2023-02-13 04:56:27,412 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=137 (was 113) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=3824 (was 4314) 2023-02-13 04:56:27,419 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=137, ProcessCount=168, AvailableMemoryMB=3824 2023-02-13 04:56:27,420 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-13 04:56:27,420 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-02-13 04:56:27,430 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1034070156:84 , srv1086859244:84 , srv110509034:84 , srv1164695387:84 , srv1474827578:84 , srv1582391675:84 , srv1655451457:84 , srv1668912712:84 , srv1697924456:84 , srv1763413503:84 , srv1769801132:84 , srv1997248638:84 , srv1999247076:84 , srv2049806587:84 , srv2050277043:84 , srv2103460414:84 , srv260639649:84 , srv285436906:84 , srv395470424:84 , srv431181120:84 , srv447148103:84 , srv459002218:84 , srv664583548:84 , srv717632540:84 , srv763249004:84 , srv848631368:84 , srv895194877:84 , srv905366796:84 , srv931667431:84 , srv1112911286:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-02-13 04:56:27,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 04:56:27,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv110509034=2, srv459002218=22, srv1086859244=1, srv1112911286=3, srv395470424=19, srv1034070156=0, srv285436906=18, srv717632540=24, srv447148103=21, srv895194877=27, srv1474827578=5, srv1668912712=8, srv2103460414=16, srv1164695387=4, srv848631368=26, srv931667431=29, srv431181120=20, srv905366796=28, srv1697924456=9, srv260639649=17, srv763249004=25, srv1655451457=7, srv1999247076=13, srv1769801132=11, srv2050277043=15, srv1582391675=6, srv664583548=23, srv2049806587=14, srv1997248638=12, srv1763413503=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 04:56:27,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:56:27,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 04:56:27,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 04:56:27,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 04:56:27,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 04:56:27,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 04:56:27,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 04:56:27,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 04:56:27,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-13 04:56:27,460 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1545006522223569, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:56:27,461 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912679869774521, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1545006522223569, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-02-13 04:58:27,446 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6254844 different iterations. Found a solution that moves 1866 regions; Going from a computed imbalance of 0.9912679869774521 to a new imbalance of 4.3553370884861656E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6911111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:58:27,453 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1034070156:90 , srv1086859244:90 , srv110509034:90 , srv1112911286:90 , srv1164695387:90 , srv1474827578:90 , srv1582391675:90 , srv1655451457:90 , srv1668912712:90 , srv1697924456:90 , srv1763413503:90 , srv1769801132:90 , srv1997248638:90 , srv1999247076:90 , srv2049806587:90 , srv2050277043:90 , srv2103460414:90 , srv260639649:90 , srv285436906:90 , srv395470424:90 , srv431181120:90 , srv447148103:90 , srv459002218:90 , srv664583548:90 , srv717632540:90 , srv763249004:90 , srv848631368:90 , srv895194877:90 , srv905366796:90 , srv931667431:90 } 2023-02-13 04:58:27,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 04:58:27,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv110509034=2, srv459002218=22, srv1086859244=1, srv1112911286=3, srv395470424=19, srv1034070156=0, srv285436906=18, srv717632540=24, srv447148103=21, srv895194877=27, srv1474827578=5, srv1668912712=8, srv2103460414=16, srv1164695387=4, srv848631368=26, srv931667431=29, srv431181120=20, srv905366796=28, srv1697924456=9, srv260639649=17, srv763249004=25, srv1655451457=7, srv1999247076=13, srv1769801132=11, srv2050277043=15, srv1582391675=6, srv664583548=23, srv2049806587=14, srv1997248638=12, srv1763413503=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 04:58:27,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 04:58:27,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-13 04:58:27,472 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 04:58:27,472 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1034070156:90 , srv1086859244:90 , srv110509034:90 , srv1112911286:90 , srv1164695387:90 , srv1474827578:90 , srv1582391675:90 , srv1655451457:90 , srv1668912712:90 , srv1697924456:90 , srv1763413503:90 , srv1769801132:90 , srv1997248638:90 , srv1999247076:90 , srv2049806587:90 , srv2050277043:90 , srv2103460414:90 , srv260639649:90 , srv285436906:90 , srv395470424:90 , srv431181120:90 , srv447148103:90 , srv459002218:90 , srv664583548:90 , srv717632540:90 , srv763249004:90 , srv848631368:90 , srv895194877:90 , srv905366796:90 , srv931667431:90 } 2023-02-13 04:58:27,484 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=167 (was 137) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=3953 (was 3824) - AvailableMemoryMB LEAK? -