2023-05-24 17:00:25,227 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-24 17:00:25,474 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-24 17:00:25,570 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-24 17:00:25,627 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-24 17:00:25,635 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-24 17:00:25,649 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=121, ProcessCount=166, AvailableMemoryMB=10389 2023-05-24 17:00:25,651 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-24 17:00:25,652 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-24 17:00:25,804 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010049033:84 , srv1019744790:84 , srv1021634646:84 , srv1022713792:84 , srv1047636174:84 , srv1074455251:84 , srv1086625969:84 , srv1129357914:84 , srv1134864315:84 , srv1246636286:84 , srv1257215191:84 , srv1266426277:84 , srv1304379788:84 , srv1321345144:84 , srv1324042170:84 , srv1328527860:84 , srv134007003:84 , srv137373037:84 , srv1377844903:84 , srv1377963148:84 , srv1378818118:84 , srv1381956602:84 , srv1396318415:84 , srv1405450041:84 , srv1407387197:84 , srv1425940653:84 , srv1443235045:84 , srv1462552322:84 , srv1462932710:84 , srv1474295372:84 , srv153602166:84 , srv1548437382:84 , srv1551226232:84 , srv15813388:84 , srv159457657:84 , srv1606966410:84 , srv1651536539:84 , srv166690709:84 , srv166789679:84 , srv1681315217:84 , srv1689318578:84 , srv1713756693:84 , srv1714950169:84 , srv1752260948:84 , srv1752506383:84 , srv1757686998:84 , srv1764381416:84 , srv1769978413:84 , srv1771320897:84 , srv182033530:84 , srv1887788471:84 , srv1930386665:84 , srv19510211:84 , srv1956846876:84 , srv1990981049:84 , srv2001357389:84 , srv2009283544:84 , srv2068000115:84 , srv2076255456:84 , srv2080397215:84 , srv2092573210:84 , srv2122168809:84 , srv224365135:84 , srv237224094:84 , srv264134886:84 , srv272347207:84 , srv290198909:84 , srv321639631:84 , srv333251368:84 , srv341377697:84 , srv372372281:84 , srv399048398:84 , srv425836047:84 , srv511279273:84 , srv555471426:84 , srv566001044:84 , srv595155424:84 , srv596352091:84 , srv617218529:84 , srv618545948:84 , srv638934668:84 , srv674564426:84 , srv708233557:84 , srv713187712:84 , srv758366338:84 , srv781071516:84 , srv808320690:84 , srv819101339:84 , srv829419926:84 , srv8455717:84 , srv860114753:84 , srv896307241:84 , srv908576215:84 , srv936618876:84 , srv948848814:84 , srv967200378:84 , srv979910108:84 , srv986976222:84 , srv991694769:84 , srv1264328930:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-24 17:00:25,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-24 17:00:25,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv896307241=92, srv1377963148=20, srv979910108=97, srv224365135=63, srv237224094=64, srv758366338=85, srv2001357389=56, srv2122168809=62, srv1134864315=8, srv1377844903=19, srv1651536539=37, srv617218529=79, srv2092573210=61, srv1328527860=16, srv1606966410=36, srv1551226232=33, srv1752506383=45, srv1047636174=4, srv2080397215=60, srv1425940653=26, srv1022713792=3, srv967200378=96, srv1956846876=54, srv1474295372=30, srv1407387197=25, srv1074455251=5, srv936618876=94, srv1266426277=12, srv1021634646=2, srv166789679=39, srv986976222=98, srv1713756693=42, srv1321345144=14, srv808320690=87, srv908576215=93, srv1769978413=48, srv991694769=99, srv860114753=91, srv264134886=65, srv1264328930=11, srv1304379788=13, srv595155424=77, srv166690709=38, srv708233557=83, srv341377697=70, srv137373037=18, srv1681315217=40, srv1246636286=9, srv1887788471=51, srv2076255456=59, srv618545948=80, srv134007003=17, srv1930386665=52, srv272347207=66, srv333251368=69, srv2068000115=58, srv1764381416=47, srv1129357914=7, srv948848814=95, srv1010049033=0, srv713187712=84, srv1443235045=27, srv1689318578=41, srv555471426=75, srv781071516=86, srv1714950169=43, srv1462932710=29, srv1396318415=23, srv321639631=68, srv674564426=82, srv1462552322=28, srv829419926=89, srv159457657=35, srv1019744790=1, srv153602166=31, srv1771320897=49, srv372372281=71, srv19510211=53, srv1990981049=55, srv596352091=78, srv1752260948=44, srv1324042170=15, srv425836047=73, srv819101339=88, srv1381956602=22, srv1086625969=6, srv290198909=67, srv566001044=76, srv1405450041=24, srv1257215191=10, srv15813388=34, srv1548437382=32, srv1757686998=46, srv2009283544=57, srv511279273=74, srv638934668=81, srv8455717=90, srv182033530=50, srv399048398=72, srv1378818118=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-24 17:00:25,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-24 17:00:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-24 17:00:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-24 17:00:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-24 17:00:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-24 17:00:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-24 17:00:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-24 17:00:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-24 17:00:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-24 17:00:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-24 17:00:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-24 17:00:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-24 17:00:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-24 17:00:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-24 17:00:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-24 17:00:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-24 17:00:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-24 17:00:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-24 17:00:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-24 17:00:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-24 17:00:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-24 17:00:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-24 17:00:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-24 17:00:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-24 17:00:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-24 17:00:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-24 17:00:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-24 17:00:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-24 17:00:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-24 17:00:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-24 17:00:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-24 17:00:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-24 17:00:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-24 17:00:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-24 17:00:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-24 17:00:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-24 17:00:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-24 17:00:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-24 17:00:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-24 17:00:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-24 17:00:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-24 17:00:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-24 17:00:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-24 17:00:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-24 17:00:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-24 17:00:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-24 17:00:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-24 17:00:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-24 17:00:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-24 17:00:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-24 17:00:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-24 17:00:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-24 17:00:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-24 17:00:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-24 17:00:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-24 17:00:25,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-24 17:00:25,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-24 17:00:25,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-24 17:00:25,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-24 17:00:25,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-24 17:00:25,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-24 17:00:25,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-24 17:00:25,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-24 17:00:25,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-24 17:00:25,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-24 17:00:25,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-24 17:00:25,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-24 17:00:25,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-24 17:00:25,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-24 17:00:25,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-24 17:00:25,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-24 17:00:25,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-24 17:00:25,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-24 17:00:25,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-24 17:00:25,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-24 17:00:25,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-24 17:00:25,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-24 17:00:25,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-24 17:00:25,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-24 17:00:25,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-24 17:00:25,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-24 17:00:25,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-24 17:00:25,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-24 17:00:25,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-24 17:00:25,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-24 17:00:25,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-24 17:00:25,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-24 17:00:25,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-24 17:00:25,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-24 17:00:25,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-24 17:00:25,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-24 17:00:25,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-24 17:00:25,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-24 17:00:25,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-24 17:00:25,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-24 17:00:25,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-24 17:00:25,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-24 17:00:25,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-24 17:00:25,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-24 17:00:25,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-24 17:00:25,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-24 17:00:26,043 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:00:26,044 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-24 17:00:26,044 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-24 17:00:26,537 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 613 ms to try 5000 different iterations. Found a solution that moves 4690 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17651177376620777. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04080313556950347, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056841678643196586, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5211111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04080313556950347, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:00:26,551 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1246636286:82 , srv1752260948:82 , srv986976222:82 , srv1304379788:83 , srv399048398:83 , srv713187712:83 , srv948848814:83 , srv1377963148:84 , srv1462932710:84 , srv166690709:84 , srv1757686998:84 , srv182033530:84 , srv1956846876:84 , srv2092573210:84 , srv272347207:84 , srv341377697:84 , srv674564426:84 , srv908576215:84 , srv991694769:84 , srv1021634646:85 , srv1086625969:85 , srv1129357914:85 , srv1134864315:85 , srv1377844903:85 , srv1381956602:85 , srv1396318415:85 , srv1407387197:85 , srv1548437382:85 , srv1681315217:85 , srv2009283544:85 , srv2122168809:85 , srv237224094:85 , srv321639631:85 , srv372372281:85 , srv566001044:85 , srv8455717:85 , srv860114753:85 , srv979910108:85 , srv1328527860:86 , srv137373037:86 , srv166789679:86 , srv1689318578:86 , srv224365135:86 , srv264134886:86 , srv290198909:86 , srv333251368:86 , srv425836047:86 , srv596352091:86 , srv936618876:86 , srv1266426277:87 , srv1321345144:87 , srv134007003:87 , srv1378818118:87 , srv153602166:87 , srv1551226232:87 , srv15813388:87 , srv159457657:87 , srv1606966410:87 , srv1713756693:87 , srv1714950169:87 , srv1769978413:87 , srv19510211:87 , srv2080397215:87 , srv555471426:87 , srv617218529:87 , srv618545948:87 , srv758366338:87 , srv808320690:87 , srv896307241:87 , srv967200378:87 , srv1019744790:88 , srv1022713792:88 , srv1047636174:88 , srv1257215191:88 , srv1405450041:88 , srv1443235045:88 , srv1462552322:88 , srv1474295372:88 , srv1651536539:88 , srv1764381416:88 , srv1930386665:88 , srv1990981049:88 , srv2001357389:88 , srv2068000115:88 , srv595155424:88 , srv638934668:88 , srv708233557:88 , srv829419926:88 , srv1752506383:89 , srv1887788471:89 , srv511279273:89 , srv781071516:89 , srv1324042170:90 , srv1425940653:90 , srv1771320897:90 , srv2076255456:90 , srv1010049033:91 , srv1074455251:91 , srv819101339:91 , srv1264328930:453 } 2023-05-24 17:00:26,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-24 17:00:26,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv896307241=92, srv1377963148=20, srv979910108=97, srv224365135=63, srv237224094=64, srv758366338=85, srv2001357389=56, srv2122168809=62, srv1134864315=8, srv1377844903=19, srv1651536539=37, srv617218529=79, srv2092573210=61, srv1328527860=16, srv1606966410=36, srv1551226232=33, srv1752506383=45, srv1047636174=4, srv2080397215=60, srv1425940653=26, srv1022713792=3, srv967200378=96, srv1956846876=54, srv1474295372=30, srv1407387197=25, srv1074455251=5, srv936618876=94, srv1266426277=12, srv1021634646=2, srv166789679=39, srv986976222=98, srv1713756693=42, srv1321345144=14, srv808320690=87, srv908576215=93, srv1769978413=48, srv991694769=99, srv860114753=91, srv264134886=65, srv1264328930=11, srv1304379788=13, srv595155424=77, srv166690709=38, srv708233557=83, srv341377697=70, srv137373037=18, srv1681315217=40, srv1246636286=9, srv1887788471=51, srv2076255456=59, srv618545948=80, srv134007003=17, srv1930386665=52, srv272347207=66, srv333251368=69, srv2068000115=58, srv1764381416=47, srv1129357914=7, srv948848814=95, srv1010049033=0, srv713187712=84, srv1443235045=27, srv1689318578=41, srv555471426=75, srv781071516=86, srv1714950169=43, srv1462932710=29, srv1396318415=23, srv321639631=68, srv674564426=82, srv1462552322=28, srv829419926=89, srv159457657=35, srv1019744790=1, srv153602166=31, srv1771320897=49, srv372372281=71, srv19510211=53, srv1990981049=55, srv596352091=78, srv1752260948=44, srv1324042170=15, srv425836047=73, srv819101339=88, srv1381956602=22, srv1086625969=6, srv290198909=67, srv566001044=76, srv1405450041=24, srv1257215191=10, srv15813388=34, srv1548437382=32, srv1757686998=46, srv2009283544=57, srv511279273=74, srv638934668=81, srv8455717=90, srv182033530=50, srv399048398=72, srv1378818118=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-24 17:00:26,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-24 17:00:26,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-24 17:00:26,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-24 17:00:26,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-24 17:00:26,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-24 17:00:26,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-24 17:00:26,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-24 17:00:26,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-24 17:00:26,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-24 17:00:26,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-24 17:00:26,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-24 17:00:26,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-24 17:00:26,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-24 17:00:26,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-24 17:00:26,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-24 17:00:26,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-24 17:00:26,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-24 17:00:26,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-24 17:00:26,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-24 17:00:26,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-24 17:00:26,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-24 17:00:26,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-24 17:00:26,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-24 17:00:26,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-24 17:00:26,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-24 17:00:26,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-24 17:00:26,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-24 17:00:26,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-24 17:00:26,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-24 17:00:26,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-24 17:00:26,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-24 17:00:26,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-24 17:00:26,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-24 17:00:26,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-24 17:00:26,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-24 17:00:26,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-24 17:00:26,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-24 17:00:26,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-24 17:00:26,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-24 17:00:26,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-24 17:00:26,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-24 17:00:26,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-24 17:00:26,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-24 17:00:26,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-24 17:00:26,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-24 17:00:26,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-24 17:00:26,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-24 17:00:26,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-24 17:00:26,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-24 17:00:26,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-24 17:00:26,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-24 17:00:26,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-24 17:00:26,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-24 17:00:26,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-24 17:00:26,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-24 17:00:26,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-24 17:00:26,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-24 17:00:26,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-24 17:00:26,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-24 17:00:26,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-24 17:00:26,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-24 17:00:26,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-24 17:00:26,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-24 17:00:26,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-24 17:00:26,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-24 17:00:26,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-24 17:00:26,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-24 17:00:26,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-24 17:00:26,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-24 17:00:26,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-24 17:00:26,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-24 17:00:26,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-24 17:00:26,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-24 17:00:26,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-24 17:00:26,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-24 17:00:26,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-24 17:00:26,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-24 17:00:26,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-24 17:00:26,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-24 17:00:26,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-24 17:00:26,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-24 17:00:26,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-24 17:00:26,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-24 17:00:26,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-24 17:00:26,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-24 17:00:26,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-24 17:00:26,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-24 17:00:26,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-24 17:00:26,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-24 17:00:26,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-24 17:00:26,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-24 17:00:26,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-24 17:00:26,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-24 17:00:26,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-24 17:00:26,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-24 17:00:26,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-24 17:00:26,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-24 17:00:26,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-24 17:00:26,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-24 17:00:26,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-24 17:00:26,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-24 17:00:26,731 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04080313556950347, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056841678643196586, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04080313556950347, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:00:26,731 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-24 17:00:26,732 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17647893368430262, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04080313556950347, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056841678643196586, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04080313556950347, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-24 17:00:26,925 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 308 ms to try 5000 different iterations. Found a solution that moves 2666 regions; Going from a computed imbalance of 0.17647893368430262 to a new imbalance of 0.015014108953314807. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021384898399012836); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035082074683309386, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2962222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021384898399012836); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:00:26,933 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1010049033:80 , srv1771320897:81 , srv1257215191:82 , srv1266426277:82 , srv1324042170:82 , srv1405450041:82 , srv1474295372:82 , srv1752506383:82 , srv2076255456:82 , srv1019744790:83 , srv1074455251:83 , srv137373037:83 , srv1887788471:83 , srv1990981049:83 , srv2001357389:83 , srv2068000115:83 , srv2080397215:83 , srv333251368:83 , srv511279273:83 , srv595155424:83 , srv638934668:83 , srv781071516:83 , srv1022713792:84 , srv1321345144:84 , srv1425940653:84 , srv1443235045:84 , srv1764381416:84 , srv2122168809:84 , srv708233557:84 , srv758366338:84 , srv829419926:84 , srv896307241:84 , srv1047636174:85 , srv1396318415:85 , srv159457657:85 , srv166789679:85 , srv1689318578:85 , srv1930386665:85 , srv290198909:85 , srv555471426:85 , srv566001044:85 , srv617218529:85 , srv808320690:85 , srv819101339:85 , srv967200378:85 , srv134007003:86 , srv1377844903:86 , srv1381956602:86 , srv1462552322:86 , srv153602166:86 , srv1551226232:86 , srv1606966410:86 , srv1651536539:86 , srv1714950169:86 , srv19510211:86 , srv2009283544:86 , srv264134886:86 , srv341377697:86 , srv596352091:86 , srv908576215:86 , srv936618876:86 , srv1086625969:87 , srv1129357914:87 , srv1304379788:87 , srv1378818118:87 , srv15813388:87 , srv166690709:87 , srv1713756693:87 , srv1769978413:87 , srv237224094:87 , srv321639631:87 , srv618545948:87 , srv8455717:87 , srv1021634646:88 , srv1134864315:88 , srv1328527860:88 , srv1377963148:88 , srv1407387197:88 , srv1462932710:88 , srv1548437382:88 , srv1757686998:88 , srv2092573210:88 , srv372372281:88 , srv425836047:88 , srv674564426:88 , srv979910108:88 , srv1246636286:89 , srv1956846876:89 , srv224365135:89 , srv399048398:89 , srv860114753:89 , srv948848814:89 , srv991694769:89 , srv1681315217:90 , srv1752260948:90 , srv182033530:90 , srv713187712:90 , srv272347207:91 , srv986976222:92 , srv1264328930:511 } 2023-05-24 17:00:26,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-24 17:00:26,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv896307241=92, srv1377963148=20, srv979910108=97, srv224365135=63, srv237224094=64, srv758366338=85, srv2001357389=56, srv2122168809=62, srv1134864315=8, srv1377844903=19, srv1651536539=37, srv617218529=79, srv2092573210=61, srv1328527860=16, srv1606966410=36, srv1551226232=33, srv1752506383=45, srv1047636174=4, srv2080397215=60, srv1425940653=26, srv1022713792=3, srv967200378=96, srv1956846876=54, srv1474295372=30, srv1407387197=25, srv1074455251=5, srv936618876=94, srv1266426277=12, srv1021634646=2, srv166789679=39, srv986976222=98, srv1713756693=42, srv1321345144=14, srv808320690=87, srv908576215=93, srv1769978413=48, srv991694769=99, srv860114753=91, srv264134886=65, srv1264328930=11, srv1304379788=13, srv595155424=77, srv166690709=38, srv708233557=83, srv341377697=70, srv137373037=18, srv1681315217=40, srv1246636286=9, srv1887788471=51, srv2076255456=59, srv618545948=80, srv134007003=17, srv1930386665=52, srv272347207=66, srv333251368=69, srv2068000115=58, srv1764381416=47, srv1129357914=7, srv948848814=95, srv1010049033=0, srv713187712=84, srv1443235045=27, srv1689318578=41, srv555471426=75, srv781071516=86, srv1714950169=43, srv1462932710=29, srv1396318415=23, srv321639631=68, srv674564426=82, srv1462552322=28, srv829419926=89, srv159457657=35, srv1019744790=1, srv153602166=31, srv1771320897=49, srv372372281=71, srv19510211=53, srv1990981049=55, srv596352091=78, srv1752260948=44, srv1324042170=15, srv425836047=73, srv819101339=88, srv1381956602=22, srv1086625969=6, srv290198909=67, srv566001044=76, srv1405450041=24, srv1257215191=10, srv15813388=34, srv1548437382=32, srv1757686998=46, srv2009283544=57, srv511279273=74, srv638934668=81, srv8455717=90, srv182033530=50, srv399048398=72, srv1378818118=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-24 17:00:26,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-24 17:00:26,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-24 17:00:26,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-24 17:00:26,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-24 17:00:26,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-24 17:00:26,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-24 17:00:26,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-24 17:00:26,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-24 17:00:26,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-24 17:00:26,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-24 17:00:26,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-24 17:00:26,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-24 17:00:26,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-24 17:00:26,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-24 17:00:26,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-24 17:00:26,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-24 17:00:26,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-24 17:00:26,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-24 17:00:26,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-24 17:00:26,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-24 17:00:26,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-24 17:00:26,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-24 17:00:26,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-24 17:00:26,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-24 17:00:26,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-24 17:00:26,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-24 17:00:26,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-24 17:00:26,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-24 17:00:26,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-24 17:00:26,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-24 17:00:26,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-24 17:00:26,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-24 17:00:26,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-24 17:00:26,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-24 17:00:26,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-24 17:00:26,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-24 17:00:26,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-24 17:00:26,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-24 17:00:26,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-24 17:00:26,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-24 17:00:26,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-24 17:00:26,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-24 17:00:26,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-24 17:00:26,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-24 17:00:26,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-24 17:00:27,052 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021384898399012836); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035082074683309386, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021384898399012836); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:00:27,052 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-24 17:00:27,052 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014995441222321394, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021384898399012836); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035082074683309386, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021384898399012836); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-24 17:00:27,213 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 244 ms to try 5000 different iterations. Found a solution that moves 1020 regions; Going from a computed imbalance of 0.014995441222321394 to a new imbalance of 8.328010063643414E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007439325174185004); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.013796663498152624); MoveCostFunction : (multiplier=7.0, imbalance=0.11333333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007439325174185004); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:00:27,215 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1681315217:82 , srv224365135:82 , srv819101339:82 , srv986976222:82 , srv1074455251:83 , srv1425940653:83 , srv15813388:83 , srv182033530:83 , srv272347207:83 , srv425836047:83 , srv860114753:83 , srv1134864315:84 , srv1328527860:84 , srv134007003:84 , srv1378818118:84 , srv1407387197:84 , srv1462552322:84 , srv153602166:84 , srv1548437382:84 , srv1551226232:84 , srv1651536539:84 , srv1713756693:84 , srv1714950169:84 , srv1769978413:84 , srv1956846876:84 , srv372372281:84 , srv991694769:84 , srv1019744790:85 , srv1021634646:85 , srv1047636174:85 , srv1086625969:85 , srv1129357914:85 , srv1324042170:85 , srv1377963148:85 , srv1443235045:85 , srv1606966410:85 , srv1752260948:85 , srv1764381416:85 , srv1887788471:85 , srv1930386665:85 , srv19510211:85 , srv1990981049:85 , srv2076255456:85 , srv2092573210:85 , srv237224094:85 , srv264134886:85 , srv399048398:85 , srv511279273:85 , srv555471426:85 , srv596352091:85 , srv617218529:85 , srv618545948:85 , srv708233557:85 , srv713187712:85 , srv808320690:85 , srv948848814:85 , srv967200378:85 , srv979910108:85 , srv1010049033:86 , srv1022713792:86 , srv1377844903:86 , srv1381956602:86 , srv1462932710:86 , srv1474295372:86 , srv159457657:86 , srv166690709:86 , srv1752506383:86 , srv1757686998:86 , srv1771320897:86 , srv2001357389:86 , srv2009283544:86 , srv2068000115:86 , srv290198909:86 , srv674564426:86 , srv781071516:86 , srv829419926:86 , srv936618876:86 , srv1246636286:87 , srv1257215191:87 , srv1304379788:87 , srv1321345144:87 , srv1396318415:87 , srv1405450041:87 , srv166789679:87 , srv1689318578:87 , srv2080397215:87 , srv2122168809:87 , srv321639631:87 , srv341377697:87 , srv595155424:87 , srv638934668:87 , srv758366338:87 , srv8455717:87 , srv896307241:87 , srv908576215:87 , srv1266426277:88 , srv137373037:88 , srv333251368:88 , srv566001044:88 , srv1264328930:560 } 2023-05-24 17:00:27,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-24 17:00:27,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv896307241=92, srv1377963148=20, srv979910108=97, srv224365135=63, srv237224094=64, srv758366338=85, srv2001357389=56, srv2122168809=62, srv1134864315=8, srv1377844903=19, srv1651536539=37, srv617218529=79, srv2092573210=61, srv1328527860=16, srv1606966410=36, srv1551226232=33, srv1752506383=45, srv1047636174=4, srv2080397215=60, srv1425940653=26, srv1022713792=3, srv967200378=96, srv1956846876=54, srv1474295372=30, srv1407387197=25, srv1074455251=5, srv936618876=94, srv1266426277=12, srv1021634646=2, srv166789679=39, srv986976222=98, srv1713756693=42, srv1321345144=14, srv808320690=87, srv908576215=93, srv1769978413=48, srv991694769=99, srv860114753=91, srv264134886=65, srv1264328930=11, srv1304379788=13, srv595155424=77, srv166690709=38, srv708233557=83, srv341377697=70, srv137373037=18, srv1681315217=40, srv1246636286=9, srv1887788471=51, srv2076255456=59, srv618545948=80, srv134007003=17, srv1930386665=52, srv272347207=66, srv333251368=69, srv2068000115=58, srv1764381416=47, srv1129357914=7, srv948848814=95, srv1010049033=0, srv713187712=84, srv1443235045=27, srv1689318578=41, srv555471426=75, srv781071516=86, srv1714950169=43, srv1462932710=29, srv1396318415=23, srv321639631=68, srv674564426=82, srv1462552322=28, srv829419926=89, srv159457657=35, srv1019744790=1, srv153602166=31, srv1771320897=49, srv372372281=71, srv19510211=53, srv1990981049=55, srv596352091=78, srv1752260948=44, srv1324042170=15, srv425836047=73, srv819101339=88, srv1381956602=22, srv1086625969=6, srv290198909=67, srv566001044=76, srv1405450041=24, srv1257215191=10, srv15813388=34, srv1548437382=32, srv1757686998=46, srv2009283544=57, srv511279273=74, srv638934668=81, srv8455717=90, srv182033530=50, srv399048398=72, srv1378818118=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-24 17:00:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-24 17:00:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-24 17:00:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-24 17:00:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-24 17:00:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-24 17:00:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-24 17:00:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-24 17:00:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-24 17:00:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-24 17:00:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-24 17:00:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-24 17:00:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-24 17:00:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-24 17:00:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-24 17:00:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-24 17:00:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-24 17:00:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-24 17:00:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-24 17:00:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-24 17:00:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-24 17:00:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-24 17:00:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-24 17:00:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-24 17:00:27,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-24 17:00:27,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-24 17:00:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-24 17:00:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-24 17:00:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-24 17:00:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-24 17:00:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-24 17:00:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-24 17:00:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-24 17:00:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-24 17:00:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-24 17:00:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-24 17:00:27,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-24 17:00:27,317 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007439325174185004); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.013796663498152624); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007439325174185004); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:00:27,317 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-24 17:00:27,317 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.25658813711197E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007439325174185004); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.013796663498152624); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007439325174185004); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-24 17:00:27,528 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 287 ms to try 5000 different iterations. Found a solution that moves 298 regions; Going from a computed imbalance of 8.25658813711197E-4 to a new imbalance of 4.504111110389445E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.03311111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:00:27,529 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv321639631:83 , srv1021634646:84 , srv1022713792:84 , srv1047636174:84 , srv1246636286:84 , srv1321345144:84 , srv1324042170:84 , srv1328527860:84 , srv1378818118:84 , srv1396318415:84 , srv1425940653:84 , srv1462552322:84 , srv1462932710:84 , srv159457657:84 , srv1651536539:84 , srv166789679:84 , srv1689318578:84 , srv1713756693:84 , srv1757686998:84 , srv1764381416:84 , srv1769978413:84 , srv1930386665:84 , srv19510211:84 , srv224365135:84 , srv272347207:84 , srv290198909:84 , srv341377697:84 , srv566001044:84 , srv595155424:84 , srv617218529:84 , srv618545948:84 , srv638934668:84 , srv713187712:84 , srv758366338:84 , srv781071516:84 , srv829419926:84 , srv8455717:84 , srv936618876:84 , srv1010049033:85 , srv1074455251:85 , srv1086625969:85 , srv1129357914:85 , srv1134864315:85 , srv1257215191:85 , srv1304379788:85 , srv134007003:85 , srv137373037:85 , srv1377844903:85 , srv1377963148:85 , srv1381956602:85 , srv1405450041:85 , srv1407387197:85 , srv1443235045:85 , srv153602166:85 , srv1548437382:85 , srv1551226232:85 , srv15813388:85 , srv1606966410:85 , srv166690709:85 , srv1681315217:85 , srv1714950169:85 , srv1752260948:85 , srv1752506383:85 , srv182033530:85 , srv1887788471:85 , srv1956846876:85 , srv1990981049:85 , srv2001357389:85 , srv2009283544:85 , srv2068000115:85 , srv2076255456:85 , srv2080397215:85 , srv2092573210:85 , srv237224094:85 , srv264134886:85 , srv333251368:85 , srv372372281:85 , srv399048398:85 , srv425836047:85 , srv511279273:85 , srv555471426:85 , srv596352091:85 , srv674564426:85 , srv708233557:85 , srv808320690:85 , srv819101339:85 , srv860114753:85 , srv896307241:85 , srv908576215:85 , srv948848814:85 , srv967200378:85 , srv979910108:85 , srv991694769:85 , srv1019744790:86 , srv1266426277:86 , srv1474295372:86 , srv1771320897:86 , srv2122168809:86 , srv986976222:86 , srv1264328930:618 } 2023-05-24 17:00:27,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-24 17:00:27,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv896307241=92, srv1377963148=20, srv979910108=97, srv224365135=63, srv237224094=64, srv758366338=85, srv2001357389=56, srv2122168809=62, srv1134864315=8, srv1377844903=19, srv1651536539=37, srv617218529=79, srv2092573210=61, srv1328527860=16, srv1606966410=36, srv1551226232=33, srv1752506383=45, srv1047636174=4, srv2080397215=60, srv1425940653=26, srv1022713792=3, srv967200378=96, srv1956846876=54, srv1474295372=30, srv1407387197=25, srv1074455251=5, srv936618876=94, srv1266426277=12, srv1021634646=2, srv166789679=39, srv986976222=98, srv1713756693=42, srv1321345144=14, srv808320690=87, srv908576215=93, srv1769978413=48, srv991694769=99, srv860114753=91, srv264134886=65, srv1264328930=11, srv1304379788=13, srv595155424=77, srv166690709=38, srv708233557=83, srv341377697=70, srv137373037=18, srv1681315217=40, srv1246636286=9, srv1887788471=51, srv2076255456=59, srv618545948=80, srv134007003=17, srv1930386665=52, srv272347207=66, srv333251368=69, srv2068000115=58, srv1764381416=47, srv1129357914=7, srv948848814=95, srv1010049033=0, srv713187712=84, srv1443235045=27, srv1689318578=41, srv555471426=75, srv781071516=86, srv1714950169=43, srv1462932710=29, srv1396318415=23, srv321639631=68, srv674564426=82, srv1462552322=28, srv829419926=89, srv159457657=35, srv1019744790=1, srv153602166=31, srv1771320897=49, srv372372281=71, srv19510211=53, srv1990981049=55, srv596352091=78, srv1752260948=44, srv1324042170=15, srv425836047=73, srv819101339=88, srv1381956602=22, srv1086625969=6, srv290198909=67, srv566001044=76, srv1405450041=24, srv1257215191=10, srv15813388=34, srv1548437382=32, srv1757686998=46, srv2009283544=57, srv511279273=74, srv638934668=81, srv8455717=90, srv182033530=50, srv399048398=72, srv1378818118=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-24 17:00:27,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-24 17:00:27,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-24 17:00:27,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-24 17:00:27,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-24 17:00:27,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-24 17:00:27,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-24 17:00:27,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-24 17:00:27,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-24 17:00:27,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-24 17:00:27,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-24 17:00:27,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-24 17:00:27,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-24 17:00:27,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-24 17:00:27,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-24 17:00:27,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-24 17:00:27,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-24 17:00:27,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-24 17:00:27,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-24 17:00:27,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-24 17:00:27,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-24 17:00:27,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-24 17:00:27,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-24 17:00:27,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-24 17:00:27,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-24 17:00:27,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-24 17:00:27,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-24 17:00:27,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-24 17:00:27,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-24 17:00:27,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-24 17:00:27,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-24 17:00:27,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-24 17:00:27,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-24 17:00:27,625 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:00:27,626 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-24 17:00:27,626 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.295447050523066E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-24 17:00:27,773 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 218 ms to try 5000 different iterations. Found a solution that moves 35 regions; Going from a computed imbalance of 4.295447050523066E-5 to a new imbalance of 8.775722801124363E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0038888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:00:27,774 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1266426277:83 , srv1606966410:83 , srv1771320897:83 , srv618545948:83 , srv674564426:83 , srv819101339:83 , srv8455717:83 , srv896307241:83 , srv979910108:83 , srv1010049033:84 , srv1019744790:84 , srv1021634646:84 , srv1022713792:84 , srv1047636174:84 , srv1074455251:84 , srv1086625969:84 , srv1129357914:84 , srv1134864315:84 , srv1246636286:84 , srv1257215191:84 , srv1304379788:84 , srv1321345144:84 , srv1328527860:84 , srv134007003:84 , srv137373037:84 , srv1377844903:84 , srv1377963148:84 , srv1378818118:84 , srv1381956602:84 , srv1405450041:84 , srv1407387197:84 , srv1443235045:84 , srv1462552322:84 , srv1462932710:84 , srv1474295372:84 , srv153602166:84 , srv1548437382:84 , srv1551226232:84 , srv15813388:84 , srv159457657:84 , srv1651536539:84 , srv166690709:84 , srv166789679:84 , srv1681315217:84 , srv1689318578:84 , srv1713756693:84 , srv1714950169:84 , srv1752260948:84 , srv1752506383:84 , srv1757686998:84 , srv1769978413:84 , srv182033530:84 , srv1887788471:84 , srv1930386665:84 , srv19510211:84 , srv1956846876:84 , srv2001357389:84 , srv2009283544:84 , srv2068000115:84 , srv2076255456:84 , srv2080397215:84 , srv2092573210:84 , srv2122168809:84 , srv237224094:84 , srv264134886:84 , srv272347207:84 , srv321639631:84 , srv333251368:84 , srv372372281:84 , srv399048398:84 , srv425836047:84 , srv511279273:84 , srv555471426:84 , srv566001044:84 , srv595155424:84 , srv596352091:84 , srv638934668:84 , srv708233557:84 , srv713187712:84 , srv758366338:84 , srv781071516:84 , srv808320690:84 , srv829419926:84 , srv860114753:84 , srv908576215:84 , srv936618876:84 , srv948848814:84 , srv967200378:84 , srv986976222:84 , srv991694769:84 , srv1324042170:85 , srv1396318415:85 , srv1425940653:85 , srv1764381416:85 , srv1990981049:85 , srv224365135:85 , srv290198909:85 , srv341377697:85 , srv617218529:85 , srv1264328930:684 } 2023-05-24 17:00:27,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-24 17:00:27,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv896307241=92, srv1377963148=20, srv979910108=97, srv224365135=63, srv237224094=64, srv758366338=85, srv2001357389=56, srv2122168809=62, srv1134864315=8, srv1377844903=19, srv1651536539=37, srv617218529=79, srv2092573210=61, srv1328527860=16, srv1606966410=36, srv1551226232=33, srv1752506383=45, srv1047636174=4, srv2080397215=60, srv1425940653=26, srv1022713792=3, srv967200378=96, srv1956846876=54, srv1474295372=30, srv1407387197=25, srv1074455251=5, srv936618876=94, srv1266426277=12, srv1021634646=2, srv166789679=39, srv986976222=98, srv1713756693=42, srv1321345144=14, srv808320690=87, srv908576215=93, srv1769978413=48, srv991694769=99, srv860114753=91, srv264134886=65, srv1264328930=11, srv1304379788=13, srv595155424=77, srv166690709=38, srv708233557=83, srv341377697=70, srv137373037=18, srv1681315217=40, srv1246636286=9, srv1887788471=51, srv2076255456=59, srv618545948=80, srv134007003=17, srv1930386665=52, srv272347207=66, srv333251368=69, srv2068000115=58, srv1764381416=47, srv1129357914=7, srv948848814=95, srv1010049033=0, srv713187712=84, srv1443235045=27, srv1689318578=41, srv555471426=75, srv781071516=86, srv1714950169=43, srv1462932710=29, srv1396318415=23, srv321639631=68, srv674564426=82, srv1462552322=28, srv829419926=89, srv159457657=35, srv1019744790=1, srv153602166=31, srv1771320897=49, srv372372281=71, srv19510211=53, srv1990981049=55, srv596352091=78, srv1752260948=44, srv1324042170=15, srv425836047=73, srv819101339=88, srv1381956602=22, srv1086625969=6, srv290198909=67, srv566001044=76, srv1405450041=24, srv1257215191=10, srv15813388=34, srv1548437382=32, srv1757686998=46, srv2009283544=57, srv511279273=74, srv638934668=81, srv8455717=90, srv182033530=50, srv399048398=72, srv1378818118=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-24 17:00:27,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-24 17:00:27,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-24 17:00:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-24 17:00:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-24 17:00:27,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-24 17:00:27,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-24 17:00:27,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-24 17:00:27,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-24 17:00:27,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-24 17:00:27,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-24 17:00:27,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-24 17:00:27,882 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.530647563026267E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:00:27,882 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1266426277:83 , srv1606966410:83 , srv1771320897:83 , srv618545948:83 , srv674564426:83 , srv819101339:83 , srv8455717:83 , srv896307241:83 , srv979910108:83 , srv1010049033:84 , srv1019744790:84 , srv1021634646:84 , srv1022713792:84 , srv1047636174:84 , srv1074455251:84 , srv1086625969:84 , srv1129357914:84 , srv1134864315:84 , srv1246636286:84 , srv1257215191:84 , srv1304379788:84 , srv1321345144:84 , srv1328527860:84 , srv134007003:84 , srv137373037:84 , srv1377844903:84 , srv1377963148:84 , srv1378818118:84 , srv1381956602:84 , srv1405450041:84 , srv1407387197:84 , srv1443235045:84 , srv1462552322:84 , srv1462932710:84 , srv1474295372:84 , srv153602166:84 , srv1548437382:84 , srv1551226232:84 , srv15813388:84 , srv159457657:84 , srv1651536539:84 , srv166690709:84 , srv166789679:84 , srv1681315217:84 , srv1689318578:84 , srv1713756693:84 , srv1714950169:84 , srv1752260948:84 , srv1752506383:84 , srv1757686998:84 , srv1769978413:84 , srv182033530:84 , srv1887788471:84 , srv1930386665:84 , srv19510211:84 , srv1956846876:84 , srv2001357389:84 , srv2009283544:84 , srv2068000115:84 , srv2076255456:84 , srv2080397215:84 , srv2092573210:84 , srv2122168809:84 , srv237224094:84 , srv264134886:84 , srv272347207:84 , srv321639631:84 , srv333251368:84 , srv372372281:84 , srv399048398:84 , srv425836047:84 , srv511279273:84 , srv555471426:84 , srv566001044:84 , srv595155424:84 , srv596352091:84 , srv638934668:84 , srv708233557:84 , srv713187712:84 , srv758366338:84 , srv781071516:84 , srv808320690:84 , srv829419926:84 , srv860114753:84 , srv908576215:84 , srv936618876:84 , srv948848814:84 , srv967200378:84 , srv986976222:84 , srv991694769:84 , srv1324042170:85 , srv1396318415:85 , srv1425940653:85 , srv1764381416:85 , srv1990981049:85 , srv224365135:85 , srv290198909:85 , srv341377697:85 , srv617218529:85 , srv1264328930:684 } 2023-05-24 17:00:27,903 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=121 (was 121), ProcessCount=166 (was 166), AvailableMemoryMB=9906 (was 10389) 2023-05-24 17:00:27,910 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=121, ProcessCount=166, AvailableMemoryMB=9906 2023-05-24 17:00:27,911 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-24 17:00:27,911 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-24 17:00:27,924 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1251691342:84 , srv1264027508:84 , srv1295640401:84 , srv1327887580:84 , srv1360562417:84 , srv1673525933:84 , srv176518960:84 , srv1810603176:84 , srv1870993552:84 , srv1919312148:84 , srv2002830081:84 , srv2007107792:84 , srv2039500674:84 , srv2084391905:84 , srv222642346:84 , srv235741296:84 , srv339262017:84 , srv339736167:84 , srv490156128:84 , srv569188757:84 , srv619878145:84 , srv664580836:84 , srv681518920:84 , srv860516511:84 , srv871457162:84 , srv877118960:84 , srv89276709:84 , srv937678285:84 , srv95978468:84 , srv20511118:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-24 17:00:27,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-24 17:00:27,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2039500674=12, srv95978468=29, srv1295640401=2, srv20511118=13, srv860516511=24, srv339262017=17, srv1251691342=0, srv490156128=19, srv681518920=23, srv2007107792=11, srv569188757=20, srv2084391905=14, srv176518960=6, srv2002830081=10, srv339736167=18, srv1360562417=4, srv1870993552=8, srv877118960=26, srv664580836=22, srv235741296=16, srv1264027508=1, srv937678285=28, srv1673525933=5, srv1919312148=9, srv89276709=27, srv222642346=15, srv1327887580=3, srv1810603176=7, srv619878145=21, srv871457162=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-24 17:00:27,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-24 17:00:27,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-24 17:00:27,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-24 17:00:27,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-24 17:00:27,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-24 17:00:27,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-24 17:00:27,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-24 17:00:27,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-24 17:00:27,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-24 17:00:27,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-24 17:00:27,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-24 17:00:27,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-24 17:00:27,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-24 17:00:27,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-24 17:00:27,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-24 17:00:27,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-24 17:00:27,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-24 17:00:27,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-24 17:00:27,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-24 17:00:27,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-24 17:00:27,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-24 17:00:27,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-24 17:00:27,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-24 17:00:27,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-24 17:00:27,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-24 17:00:27,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-24 17:00:27,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-24 17:00:27,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-24 17:00:27,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-24 17:00:27,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-24 17:00:27,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-24 17:00:27,964 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0668136752589077, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:00:27,964 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912403570973355, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0668136752589077, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-24 17:02:27,944 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9978080 different iterations. Found a solution that moves 1839 regions; Going from a computed imbalance of 0.9912403570973355 to a new imbalance of 4.292317741546655E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6811111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:02:27,951 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1251691342:90 , srv1264027508:90 , srv1295640401:90 , srv1327887580:90 , srv1360562417:90 , srv1673525933:90 , srv176518960:90 , srv1810603176:90 , srv1870993552:90 , srv1919312148:90 , srv2002830081:90 , srv2007107792:90 , srv2039500674:90 , srv20511118:90 , srv2084391905:90 , srv222642346:90 , srv235741296:90 , srv339262017:90 , srv339736167:90 , srv490156128:90 , srv569188757:90 , srv619878145:90 , srv664580836:90 , srv681518920:90 , srv860516511:90 , srv871457162:90 , srv877118960:90 , srv89276709:90 , srv937678285:90 , srv95978468:90 } 2023-05-24 17:02:27,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-24 17:02:27,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2039500674=12, srv95978468=29, srv1295640401=2, srv20511118=13, srv860516511=24, srv339262017=17, srv1251691342=0, srv490156128=19, srv681518920=23, srv2007107792=11, srv569188757=20, srv2084391905=14, srv176518960=6, srv2002830081=10, srv339736167=18, srv1360562417=4, srv1870993552=8, srv877118960=26, srv664580836=22, srv235741296=16, srv1264027508=1, srv937678285=28, srv1673525933=5, srv1919312148=9, srv89276709=27, srv222642346=15, srv1327887580=3, srv1810603176=7, srv619878145=21, srv871457162=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-24 17:02:27,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-24 17:02:27,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-24 17:02:27,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-24 17:02:27,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-24 17:02:27,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-24 17:02:27,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-24 17:02:27,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-24 17:02:27,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-24 17:02:27,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-24 17:02:27,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-24 17:02:27,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-24 17:02:27,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-24 17:02:27,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-24 17:02:27,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-24 17:02:27,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-24 17:02:27,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-24 17:02:27,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-24 17:02:27,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-24 17:02:27,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-24 17:02:27,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-24 17:02:27,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-24 17:02:27,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-24 17:02:27,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-24 17:02:27,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-24 17:02:27,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-24 17:02:27,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-24 17:02:27,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-24 17:02:27,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-24 17:02:27,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-24 17:02:27,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-24 17:02:27,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-24 17:02:27,973 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-24 17:02:27,973 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1251691342:90 , srv1264027508:90 , srv1295640401:90 , srv1327887580:90 , srv1360562417:90 , srv1673525933:90 , srv176518960:90 , srv1810603176:90 , srv1870993552:90 , srv1919312148:90 , srv2002830081:90 , srv2007107792:90 , srv2039500674:90 , srv20511118:90 , srv2084391905:90 , srv222642346:90 , srv235741296:90 , srv339262017:90 , srv339736167:90 , srv490156128:90 , srv569188757:90 , srv619878145:90 , srv664580836:90 , srv681518920:90 , srv860516511:90 , srv871457162:90 , srv877118960:90 , srv89276709:90 , srv937678285:90 , srv95978468:90 } 2023-05-24 17:02:27,988 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=110 (was 121), ProcessCount=164 (was 166), AvailableMemoryMB=9769 (was 9906)