2024-02-04 08:53:08,131 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-02-04 08:53:08,422 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-02-04 08:53:08,543 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-02-04 08:53:08,608 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-04 08:53:08,616 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-04 08:53:08,631 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=297, ProcessCount=167, AvailableMemoryMB=3037 2024-02-04 08:53:08,633 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-04 08:53:08,633 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-04 08:53:08,770 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1005020829:84 , srv1029616948:84 , srv1035260777:84 , srv1046129279:84 , srv1059984574:84 , srv1062434145:84 , srv1072223560:84 , srv109882336:84 , srv1128817224:84 , srv1130941437:84 , srv1131200254:84 , srv1233164660:84 , srv1253531359:84 , srv1272447793:84 , srv128055670:84 , srv1285953640:84 , srv1299484139:84 , srv1318963952:84 , srv1319347438:84 , srv1320674882:84 , srv1322944021:84 , srv1324199713:84 , srv1343517547:84 , srv1373084045:84 , srv1380065610:84 , srv1396362142:84 , srv1407502069:84 , srv1489666929:84 , srv1491111121:84 , srv1498894790:84 , srv1500059855:84 , srv1505310245:84 , srv1530286860:84 , srv1551264063:84 , srv1553534104:84 , srv1554482905:84 , srv1590904846:84 , srv1600189918:84 , srv1603041418:84 , srv164829939:84 , srv166816278:84 , srv1685605944:84 , srv1730486610:84 , srv1742085343:84 , srv1755624829:84 , srv1757522971:84 , srv1760826513:84 , srv1764244912:84 , srv1770073500:84 , srv1782211823:84 , srv178935620:84 , srv1801342508:84 , srv1817742175:84 , srv1837021797:84 , srv1839282699:84 , srv1853961011:84 , srv1867617233:84 , srv1893039794:84 , srv1925358928:84 , srv1945499319:84 , srv1961749353:84 , srv1964262505:84 , srv2104779090:84 , srv2110409661:84 , srv2114135568:84 , srv2125069462:84 , srv23822659:84 , srv241630345:84 , srv271153800:84 , srv271223098:84 , srv295731606:84 , srv32648829:84 , srv332021147:84 , srv346292121:84 , srv351529461:84 , srv42114356:84 , srv424191789:84 , srv472470690:84 , srv51357780:84 , srv567641530:84 , srv570622890:84 , srv622181267:84 , srv629492384:84 , srv644323326:84 , srv680148768:84 , srv687392325:84 , srv722405515:84 , srv74580547:84 , srv756278990:84 , srv786095946:84 , srv802773292:84 , srv813170652:84 , srv856933879:84 , srv859397182:84 , srv870910835:84 , srv892910456:84 , srv899263376:84 , srv95953121:84 , srv959558902:84 , srv1040572698:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-02-04 08:53:08,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-04 08:53:08,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv622181267=82, srv1491111121=29, srv2104779090=63, srv1072223560=7, srv109882336=8, srv1319347438=19, srv1005020829=0, srv629492384=83, srv1893039794=58, srv1131200254=11, srv1530286860=33, srv1590904846=37, srv813170652=92, srv1554482905=36, srv1059984574=5, srv1233164660=12, srv1500059855=31, srv1498894790=30, srv1553534104=35, srv1964262505=62, srv959558902=99, srv722405515=87, srv128055670=15, srv1867617233=57, srv1782211823=50, srv1396362142=26, srv1730486610=43, srv756278990=89, srv1505310245=32, srv271153800=69, srv424191789=77, srv332021147=73, srv680148768=85, srv346292121=74, srv1600189918=38, srv95953121=98, srv1373084045=24, srv687392325=86, srv23822659=67, srv786095946=90, srv1770073500=49, srv1040572698=3, srv1029616948=1, srv42114356=76, srv644323326=84, srv241630345=68, srv178935620=51, srv1817742175=53, srv51357780=79, srv1603041418=39, srv1757522971=46, srv1961749353=61, srv1299484139=17, srv899263376=97, srv271223098=70, srv1760826513=47, srv1035260777=2, srv1407502069=27, srv1062434145=6, srv295731606=71, srv1925358928=59, srv802773292=91, srv1130941437=10, srv164829939=40, srv1853961011=56, srv1801342508=52, srv892910456=96, srv1320674882=20, srv870910835=95, srv1380065610=25, srv1324199713=22, srv1489666929=28, srv1272447793=14, srv1318963952=18, srv1046129279=4, srv74580547=88, srv1128817224=9, srv472470690=78, srv1285953640=16, srv1322944021=21, srv1685605944=42, srv1551264063=34, srv2125069462=66, srv570622890=81, srv859397182=94, srv567641530=80, srv2114135568=65, srv1839282699=55, srv1945499319=60, srv1764244912=48, srv1742085343=44, srv1755624829=45, srv856933879=93, srv166816278=41, srv1837021797=54, srv32648829=72, srv351529461=75, srv1253531359=13, srv2110409661=64, srv1343517547=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-04 08:53:08,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-04 08:53:08,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-04 08:53:08,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-04 08:53:08,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-04 08:53:08,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-04 08:53:08,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-04 08:53:08,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-04 08:53:08,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-04 08:53:08,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-04 08:53:08,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-04 08:53:08,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-04 08:53:08,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-04 08:53:08,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-04 08:53:08,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-04 08:53:08,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-04 08:53:08,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-04 08:53:08,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-04 08:53:08,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-04 08:53:08,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-04 08:53:08,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-04 08:53:08,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-04 08:53:08,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-04 08:53:08,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-04 08:53:08,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-04 08:53:08,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-04 08:53:08,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-04 08:53:08,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-04 08:53:08,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-04 08:53:08,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-04 08:53:08,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-04 08:53:08,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-04 08:53:08,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-04 08:53:08,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-04 08:53:08,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-04 08:53:08,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-04 08:53:08,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-04 08:53:08,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-04 08:53:08,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-04 08:53:08,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-04 08:53:08,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-04 08:53:08,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-04 08:53:08,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-04 08:53:08,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-04 08:53:08,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-04 08:53:08,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-04 08:53:08,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-04 08:53:08,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-04 08:53:08,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-04 08:53:08,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-04 08:53:08,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-04 08:53:08,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-04 08:53:08,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-04 08:53:08,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-04 08:53:08,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-04 08:53:08,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-04 08:53:08,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-04 08:53:08,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-04 08:53:08,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-04 08:53:08,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-04 08:53:08,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-04 08:53:08,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-04 08:53:08,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-04 08:53:08,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-04 08:53:08,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-04 08:53:08,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-04 08:53:08,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-04 08:53:08,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-04 08:53:08,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-04 08:53:08,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-04 08:53:08,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-04 08:53:08,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-04 08:53:08,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-04 08:53:08,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-04 08:53:08,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-04 08:53:08,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-04 08:53:08,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-04 08:53:08,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-04 08:53:08,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-04 08:53:08,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-04 08:53:08,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-04 08:53:08,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-04 08:53:08,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-04 08:53:08,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-04 08:53:08,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-04 08:53:08,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-04 08:53:08,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-04 08:53:08,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-04 08:53:08,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-04 08:53:08,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-04 08:53:08,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-04 08:53:08,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-04 08:53:08,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-04 08:53:08,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-04 08:53:08,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-04 08:53:08,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-04 08:53:08,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-04 08:53:08,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-04 08:53:08,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-04 08:53:08,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-04 08:53:08,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-04 08:53:08,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-04 08:53:09,313 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:53:09,313 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-04 08:53:09,314 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-04 08:53:09,781 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 842 ms to try 5000 different iterations. Found a solution that moves 4701 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17855678812650178. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03997286771531045, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057021074332825265, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5223333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03997286771531045, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17391666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23883333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:53:09,798 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1324199713:81 , srv1770073500:81 , srv1867617233:81 , srv424191789:81 , srv1272447793:82 , srv241630345:82 , srv332021147:82 , srv351529461:82 , srv813170652:82 , srv1005020829:83 , srv1590904846:83 , srv2110409661:83 , srv271153800:83 , srv680148768:83 , srv856933879:83 , srv899263376:83 , srv95953121:83 , srv1131200254:84 , srv1380065610:84 , srv1498894790:84 , srv1685605944:84 , srv1742085343:84 , srv1757522971:84 , srv1764244912:84 , srv1839282699:84 , srv687392325:84 , srv1072223560:85 , srv1299484139:85 , srv1554482905:85 , srv1801342508:85 , srv1964262505:85 , srv295731606:85 , srv644323326:85 , srv892910456:85 , srv1062434145:86 , srv1253531359:86 , srv1318963952:86 , srv1373084045:86 , srv1530286860:86 , srv1551264063:86 , srv1603041418:86 , srv166816278:86 , srv1730486610:86 , srv1817742175:86 , srv1961749353:86 , srv2125069462:86 , srv74580547:86 , srv756278990:86 , srv786095946:86 , srv959558902:86 , srv1029616948:87 , srv1035260777:87 , srv1059984574:87 , srv1320674882:87 , srv1407502069:87 , srv1489666929:87 , srv1500059855:87 , srv1600189918:87 , srv1760826513:87 , srv1945499319:87 , srv2114135568:87 , srv346292121:87 , srv42114356:87 , srv472470690:87 , srv622181267:87 , srv629492384:87 , srv859397182:87 , srv1046129279:88 , srv109882336:88 , srv1128817224:88 , srv1233164660:88 , srv1285953640:88 , srv1505310245:88 , srv164829939:88 , srv1755624829:88 , srv1893039794:88 , srv2104779090:88 , srv570622890:88 , srv870910835:88 , srv1319347438:89 , srv1322944021:89 , srv1491111121:89 , srv1837021797:89 , srv23822659:89 , srv32648829:89 , srv802773292:89 , srv128055670:90 , srv1343517547:90 , srv1396362142:90 , srv1553534104:90 , srv1782211823:90 , srv271223098:90 , srv567641530:90 , srv1130941437:91 , srv1853961011:91 , srv51357780:91 , srv1925358928:92 , srv178935620:95 , srv722405515:97 , srv1040572698:445 } 2024-02-04 08:53:09,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-04 08:53:09,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv622181267=82, srv1491111121=29, srv2104779090=63, srv1072223560=7, srv109882336=8, srv1319347438=19, srv1005020829=0, srv629492384=83, srv1893039794=58, srv1131200254=11, srv1530286860=33, srv1590904846=37, srv813170652=92, srv1554482905=36, srv1059984574=5, srv1233164660=12, srv1500059855=31, srv1498894790=30, srv1553534104=35, srv1964262505=62, srv959558902=99, srv722405515=87, srv128055670=15, srv1867617233=57, srv1782211823=50, srv1396362142=26, srv1730486610=43, srv756278990=89, srv1505310245=32, srv271153800=69, srv424191789=77, srv332021147=73, srv680148768=85, srv346292121=74, srv1600189918=38, srv95953121=98, srv1373084045=24, srv687392325=86, srv23822659=67, srv786095946=90, srv1770073500=49, srv1040572698=3, srv1029616948=1, srv42114356=76, srv644323326=84, srv241630345=68, srv178935620=51, srv1817742175=53, srv51357780=79, srv1603041418=39, srv1757522971=46, srv1961749353=61, srv1299484139=17, srv899263376=97, srv271223098=70, srv1760826513=47, srv1035260777=2, srv1407502069=27, srv1062434145=6, srv295731606=71, srv1925358928=59, srv802773292=91, srv1130941437=10, srv164829939=40, srv1853961011=56, srv1801342508=52, srv892910456=96, srv1320674882=20, srv870910835=95, srv1380065610=25, srv1324199713=22, srv1489666929=28, srv1272447793=14, srv1318963952=18, srv1046129279=4, srv74580547=88, srv1128817224=9, srv472470690=78, srv1285953640=16, srv1322944021=21, srv1685605944=42, srv1551264063=34, srv2125069462=66, srv570622890=81, srv859397182=94, srv567641530=80, srv2114135568=65, srv1839282699=55, srv1945499319=60, srv1764244912=48, srv1742085343=44, srv1755624829=45, srv856933879=93, srv166816278=41, srv1837021797=54, srv32648829=72, srv351529461=75, srv1253531359=13, srv2110409661=64, srv1343517547=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-04 08:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-04 08:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-04 08:53:09,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-04 08:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-04 08:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-04 08:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-04 08:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-04 08:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-04 08:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-04 08:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-04 08:53:09,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-04 08:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-04 08:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-04 08:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-04 08:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-04 08:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-04 08:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-04 08:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-04 08:53:09,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-04 08:53:09,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-04 08:53:09,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-04 08:53:09,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-04 08:53:09,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-04 08:53:09,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-04 08:53:09,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-04 08:53:09,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-04 08:53:09,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-04 08:53:09,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-04 08:53:09,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-04 08:53:09,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-04 08:53:09,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-04 08:53:09,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-04 08:53:09,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-04 08:53:09,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-04 08:53:09,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-04 08:53:09,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-04 08:53:09,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-04 08:53:09,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-04 08:53:09,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-04 08:53:09,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-04 08:53:09,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-04 08:53:09,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-04 08:53:09,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-04 08:53:09,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-04 08:53:09,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-04 08:53:09,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-04 08:53:09,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-04 08:53:09,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-04 08:53:09,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-04 08:53:09,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-04 08:53:09,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-04 08:53:09,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-04 08:53:09,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-04 08:53:09,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-04 08:53:09,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-04 08:53:09,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-04 08:53:09,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-04 08:53:09,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-04 08:53:09,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-04 08:53:09,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-04 08:53:09,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-04 08:53:09,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-04 08:53:09,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-04 08:53:09,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-04 08:53:09,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-04 08:53:09,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-04 08:53:09,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-04 08:53:09,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-04 08:53:09,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-04 08:53:09,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-04 08:53:09,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-04 08:53:09,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-04 08:53:09,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-04 08:53:09,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-04 08:53:09,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-04 08:53:09,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-04 08:53:09,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-04 08:53:09,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-04 08:53:09,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-04 08:53:09,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-04 08:53:09,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-04 08:53:09,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-04 08:53:09,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-04 08:53:09,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-04 08:53:09,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-04 08:53:09,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-04 08:53:09,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-04 08:53:09,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-04 08:53:09,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-04 08:53:09,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-04 08:53:09,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-04 08:53:09,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-04 08:53:09,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-04 08:53:09,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-04 08:53:09,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-04 08:53:09,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-04 08:53:09,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-04 08:53:09,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-04 08:53:09,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-04 08:53:09,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-04 08:53:09,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-04 08:53:10,014 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03997286771531045, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057021074332825265, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03997286771531045, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17391666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23883333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:53:10,015 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-04 08:53:10,015 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17852387102095035, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03997286771531045, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057021074332825265, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03997286771531045, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17391666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23883333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-04 08:53:10,339 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 454 ms to try 5000 different iterations. Found a solution that moves 2632 regions; Going from a computed imbalance of 0.17852387102095035 to a new imbalance of 0.014714770403692732. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023527332053068693); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03467663674294941, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29244444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023527332053068693); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04608333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:53:10,348 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv722405515:73 , srv1396362142:80 , srv1925358928:80 , srv51357780:80 , srv1046129279:81 , srv1322944021:81 , srv1491111121:81 , srv178935620:81 , srv1130941437:82 , srv128055670:82 , srv1285953640:82 , srv1553534104:82 , srv1755624829:82 , srv1782211823:82 , srv1837021797:82 , srv1853961011:82 , srv271223098:82 , srv32648829:82 , srv109882336:83 , srv1319347438:83 , srv1343517547:83 , srv1505310245:83 , srv1893039794:83 , srv567641530:83 , srv802773292:83 , srv1029616948:84 , srv1035260777:84 , srv1059984574:84 , srv1253531359:84 , srv1489666929:84 , srv1530286860:84 , srv1603041418:84 , srv1945499319:84 , srv23822659:84 , srv622181267:84 , srv1062434145:85 , srv1072223560:85 , srv1233164660:85 , srv1299484139:85 , srv1373084045:85 , srv1500059855:85 , srv164829939:85 , srv1760826513:85 , srv42114356:85 , srv472470690:85 , srv756278990:85 , srv856933879:85 , srv1005020829:86 , srv1128817224:86 , srv1320674882:86 , srv1407502069:86 , srv166816278:86 , srv1685605944:86 , srv1730486610:86 , srv1817742175:86 , srv2104779090:86 , srv2114135568:86 , srv295731606:86 , srv570622890:86 , srv687392325:86 , srv786095946:86 , srv859397182:86 , srv870910835:86 , srv892910456:86 , srv1380065610:87 , srv1498894790:87 , srv1551264063:87 , srv1554482905:87 , srv1600189918:87 , srv1757522971:87 , srv1801342508:87 , srv2125069462:87 , srv346292121:87 , srv644323326:87 , srv959558902:87 , srv1131200254:88 , srv1272447793:88 , srv1318963952:88 , srv1742085343:88 , srv1964262505:88 , srv2110409661:88 , srv424191789:88 , srv629492384:88 , srv680148768:88 , srv1590904846:89 , srv1770073500:89 , srv1961749353:89 , srv271153800:89 , srv95953121:89 , srv1764244912:90 , srv1839282699:90 , srv1867617233:90 , srv241630345:90 , srv351529461:90 , srv899263376:90 , srv813170652:91 , srv1324199713:92 , srv332021147:92 , srv74580547:93 , srv1040572698:538 } 2024-02-04 08:53:10,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-04 08:53:10,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv622181267=82, srv1491111121=29, srv2104779090=63, srv1072223560=7, srv109882336=8, srv1319347438=19, srv1005020829=0, srv629492384=83, srv1893039794=58, srv1131200254=11, srv1530286860=33, srv1590904846=37, srv813170652=92, srv1554482905=36, srv1059984574=5, srv1233164660=12, srv1500059855=31, srv1498894790=30, srv1553534104=35, srv1964262505=62, srv959558902=99, srv722405515=87, srv128055670=15, srv1867617233=57, srv1782211823=50, srv1396362142=26, srv1730486610=43, srv756278990=89, srv1505310245=32, srv271153800=69, srv424191789=77, srv332021147=73, srv680148768=85, srv346292121=74, srv1600189918=38, srv95953121=98, srv1373084045=24, srv687392325=86, srv23822659=67, srv786095946=90, srv1770073500=49, srv1040572698=3, srv1029616948=1, srv42114356=76, srv644323326=84, srv241630345=68, srv178935620=51, srv1817742175=53, srv51357780=79, srv1603041418=39, srv1757522971=46, srv1961749353=61, srv1299484139=17, srv899263376=97, srv271223098=70, srv1760826513=47, srv1035260777=2, srv1407502069=27, srv1062434145=6, srv295731606=71, srv1925358928=59, srv802773292=91, srv1130941437=10, srv164829939=40, srv1853961011=56, srv1801342508=52, srv892910456=96, srv1320674882=20, srv870910835=95, srv1380065610=25, srv1324199713=22, srv1489666929=28, srv1272447793=14, srv1318963952=18, srv1046129279=4, srv74580547=88, srv1128817224=9, srv472470690=78, srv1285953640=16, srv1322944021=21, srv1685605944=42, srv1551264063=34, srv2125069462=66, srv570622890=81, srv859397182=94, srv567641530=80, srv2114135568=65, srv1839282699=55, srv1945499319=60, srv1764244912=48, srv1742085343=44, srv1755624829=45, srv856933879=93, srv166816278=41, srv1837021797=54, srv32648829=72, srv351529461=75, srv1253531359=13, srv2110409661=64, srv1343517547=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-04 08:53:10,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-04 08:53:10,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-04 08:53:10,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-04 08:53:10,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-04 08:53:10,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-04 08:53:10,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-04 08:53:10,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-04 08:53:10,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-04 08:53:10,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-04 08:53:10,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-04 08:53:10,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-04 08:53:10,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-04 08:53:10,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-04 08:53:10,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-04 08:53:10,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-04 08:53:10,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-04 08:53:10,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-04 08:53:10,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-04 08:53:10,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-04 08:53:10,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-04 08:53:10,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-04 08:53:10,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-04 08:53:10,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-04 08:53:10,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-04 08:53:10,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-04 08:53:10,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-04 08:53:10,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-04 08:53:10,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-04 08:53:10,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-04 08:53:10,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-04 08:53:10,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-04 08:53:10,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-04 08:53:10,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-04 08:53:10,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-04 08:53:10,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-04 08:53:10,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-04 08:53:10,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-04 08:53:10,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-04 08:53:10,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-04 08:53:10,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-04 08:53:10,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-04 08:53:10,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-04 08:53:10,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-04 08:53:10,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-04 08:53:10,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-04 08:53:10,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-04 08:53:10,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-04 08:53:10,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-04 08:53:10,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-04 08:53:10,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-04 08:53:10,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-04 08:53:10,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-04 08:53:10,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-04 08:53:10,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-04 08:53:10,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-04 08:53:10,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-04 08:53:10,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-04 08:53:10,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-04 08:53:10,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-04 08:53:10,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-04 08:53:10,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-04 08:53:10,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-04 08:53:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-04 08:53:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-04 08:53:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-04 08:53:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-04 08:53:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-04 08:53:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-04 08:53:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-04 08:53:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-04 08:53:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-04 08:53:10,501 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023527332053068693); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03467663674294941, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023527332053068693); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04608333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:53:10,501 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-04 08:53:10,501 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014696340745787755, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023527332053068693); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03467663674294941, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023527332053068693); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04608333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-04 08:53:10,921 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 522 ms to try 5000 different iterations. Found a solution that moves 1000 regions; Going from a computed imbalance of 0.014696340745787755 to a new imbalance of 7.88701873175223E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006217570812329759); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00867158627528677); MoveCostFunction : (multiplier=7.0, imbalance=0.1111111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006217570812329759); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:53:10,923 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv74580547:80 , srv178935620:82 , srv1961749353:82 , srv1318963952:83 , srv1764244912:83 , srv1839282699:83 , srv2114135568:83 , srv570622890:83 , srv629492384:83 , srv870910835:83 , srv1128817224:84 , srv1233164660:84 , srv1551264063:84 , srv1600189918:84 , srv1817742175:84 , srv2104779090:84 , srv2125069462:84 , srv23822659:84 , srv346292121:84 , srv567641530:84 , srv813170652:84 , srv899263376:84 , srv1130941437:85 , srv128055670:85 , srv1320674882:85 , srv1324199713:85 , srv1343517547:85 , srv1407502069:85 , srv1500059855:85 , srv1554482905:85 , srv1590904846:85 , srv164829939:85 , srv166816278:85 , srv1782211823:85 , srv1853961011:85 , srv1925358928:85 , srv1964262505:85 , srv241630345:85 , srv295731606:85 , srv332021147:85 , srv351529461:85 , srv42114356:85 , srv472470690:85 , srv786095946:85 , srv802773292:85 , srv859397182:85 , srv95953121:85 , srv959558902:85 , srv1035260777:86 , srv1059984574:86 , srv1062434145:86 , srv109882336:86 , srv1131200254:86 , srv1299484139:86 , srv1319347438:86 , srv1373084045:86 , srv1380065610:86 , srv1489666929:86 , srv1498894790:86 , srv1505310245:86 , srv1553534104:86 , srv1742085343:86 , srv1801342508:86 , srv1867617233:86 , srv1893039794:86 , srv2110409661:86 , srv271153800:86 , srv271223098:86 , srv32648829:86 , srv644323326:86 , srv680148768:86 , srv1029616948:87 , srv1253531359:87 , srv1272447793:87 , srv1285953640:87 , srv1322944021:87 , srv1530286860:87 , srv1603041418:87 , srv1730486610:87 , srv1755624829:87 , srv1757522971:87 , srv1760826513:87 , srv1770073500:87 , srv1837021797:87 , srv1945499319:87 , srv51357780:87 , srv622181267:87 , srv722405515:87 , srv756278990:87 , srv1005020829:88 , srv1046129279:88 , srv1396362142:88 , srv1491111121:88 , srv1685605944:88 , srv687392325:88 , srv892910456:88 , srv1072223560:89 , srv424191789:89 , srv856933879:89 , srv1040572698:530 } 2024-02-04 08:53:10,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-04 08:53:10,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv622181267=82, srv1491111121=29, srv2104779090=63, srv1072223560=7, srv109882336=8, srv1319347438=19, srv1005020829=0, srv629492384=83, srv1893039794=58, srv1131200254=11, srv1530286860=33, srv1590904846=37, srv813170652=92, srv1554482905=36, srv1059984574=5, srv1233164660=12, srv1500059855=31, srv1498894790=30, srv1553534104=35, srv1964262505=62, srv959558902=99, srv722405515=87, srv128055670=15, srv1867617233=57, srv1782211823=50, srv1396362142=26, srv1730486610=43, srv756278990=89, srv1505310245=32, srv271153800=69, srv424191789=77, srv332021147=73, srv680148768=85, srv346292121=74, srv1600189918=38, srv95953121=98, srv1373084045=24, srv687392325=86, srv23822659=67, srv786095946=90, srv1770073500=49, srv1040572698=3, srv1029616948=1, srv42114356=76, srv644323326=84, srv241630345=68, srv178935620=51, srv1817742175=53, srv51357780=79, srv1603041418=39, srv1757522971=46, srv1961749353=61, srv1299484139=17, srv899263376=97, srv271223098=70, srv1760826513=47, srv1035260777=2, srv1407502069=27, srv1062434145=6, srv295731606=71, srv1925358928=59, srv802773292=91, srv1130941437=10, srv164829939=40, srv1853961011=56, srv1801342508=52, srv892910456=96, srv1320674882=20, srv870910835=95, srv1380065610=25, srv1324199713=22, srv1489666929=28, srv1272447793=14, srv1318963952=18, srv1046129279=4, srv74580547=88, srv1128817224=9, srv472470690=78, srv1285953640=16, srv1322944021=21, srv1685605944=42, srv1551264063=34, srv2125069462=66, srv570622890=81, srv859397182=94, srv567641530=80, srv2114135568=65, srv1839282699=55, srv1945499319=60, srv1764244912=48, srv1742085343=44, srv1755624829=45, srv856933879=93, srv166816278=41, srv1837021797=54, srv32648829=72, srv351529461=75, srv1253531359=13, srv2110409661=64, srv1343517547=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-04 08:53:10,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-04 08:53:10,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-04 08:53:10,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-04 08:53:10,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-04 08:53:10,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-04 08:53:10,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-04 08:53:10,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-04 08:53:10,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-04 08:53:10,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-04 08:53:10,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-04 08:53:10,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-04 08:53:10,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-04 08:53:10,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-04 08:53:10,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-04 08:53:10,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-04 08:53:10,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-04 08:53:10,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-04 08:53:10,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-04 08:53:10,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-04 08:53:10,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-04 08:53:10,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-04 08:53:10,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-04 08:53:10,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-04 08:53:10,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-04 08:53:10,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-04 08:53:10,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-04 08:53:10,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-04 08:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-04 08:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-04 08:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-04 08:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-04 08:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-04 08:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-04 08:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-04 08:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-04 08:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-04 08:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-04 08:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-04 08:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-04 08:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-04 08:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-04 08:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-04 08:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-04 08:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-04 08:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-04 08:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-04 08:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-04 08:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-04 08:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-04 08:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-04 08:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-04 08:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-04 08:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-04 08:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-04 08:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-04 08:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-04 08:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-04 08:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-04 08:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-04 08:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-04 08:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-04 08:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-04 08:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-04 08:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-04 08:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-04 08:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-04 08:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-04 08:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-04 08:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-04 08:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-04 08:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-04 08:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-04 08:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-04 08:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-04 08:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-04 08:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-04 08:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-04 08:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-04 08:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-04 08:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-04 08:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-04 08:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-04 08:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-04 08:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-04 08:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-04 08:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-04 08:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-04 08:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-04 08:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-04 08:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-04 08:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-04 08:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-04 08:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-04 08:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-04 08:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-04 08:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-04 08:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-04 08:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-04 08:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-04 08:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-04 08:53:10,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-04 08:53:11,096 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006217570812329759); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00867158627528677); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006217570812329759); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:53:11,096 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-04 08:53:11,097 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.816997235152775E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006217570812329759); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00867158627528677); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006217570812329759); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-04 08:53:11,399 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 432 ms to try 5000 different iterations. Found a solution that moves 256 regions; Going from a computed imbalance of 7.816997235152775E-4 to a new imbalance of 4.4181091682712206E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.028444444444444446, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:53:11,401 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1553534104:83 , srv1730486610:83 , srv1760826513:83 , srv1801342508:83 , srv271223098:83 , srv332021147:83 , srv51357780:83 , srv74580547:83 , srv892910456:83 , srv1029616948:84 , srv1072223560:84 , srv1128817224:84 , srv1130941437:84 , srv1131200254:84 , srv1319347438:84 , srv1320674882:84 , srv1324199713:84 , srv1343517547:84 , srv1396362142:84 , srv1407502069:84 , srv1491111121:84 , srv1600189918:84 , srv164829939:84 , srv166816278:84 , srv1757522971:84 , srv178935620:84 , srv1837021797:84 , srv1839282699:84 , srv1853961011:84 , srv1964262505:84 , srv2104779090:84 , srv271153800:84 , srv346292121:84 , srv424191789:84 , srv622181267:84 , srv629492384:84 , srv644323326:84 , srv687392325:84 , srv722405515:84 , srv756278990:84 , srv859397182:84 , srv959558902:84 , srv1005020829:85 , srv1046129279:85 , srv1059984574:85 , srv1062434145:85 , srv109882336:85 , srv1233164660:85 , srv1253531359:85 , srv1272447793:85 , srv128055670:85 , srv1285953640:85 , srv1318963952:85 , srv1322944021:85 , srv1373084045:85 , srv1380065610:85 , srv1489666929:85 , srv1498894790:85 , srv1500059855:85 , srv1505310245:85 , srv1530286860:85 , srv1551264063:85 , srv1554482905:85 , srv1590904846:85 , srv1603041418:85 , srv1685605944:85 , srv1742085343:85 , srv1755624829:85 , srv1764244912:85 , srv1770073500:85 , srv1867617233:85 , srv1893039794:85 , srv1925358928:85 , srv1945499319:85 , srv1961749353:85 , srv2110409661:85 , srv2125069462:85 , srv23822659:85 , srv241630345:85 , srv295731606:85 , srv32648829:85 , srv351529461:85 , srv42114356:85 , srv567641530:85 , srv570622890:85 , srv786095946:85 , srv802773292:85 , srv813170652:85 , srv856933879:85 , srv870910835:85 , srv899263376:85 , srv95953121:85 , srv1035260777:86 , srv1299484139:86 , srv1782211823:86 , srv1817742175:86 , srv2114135568:86 , srv472470690:86 , srv680148768:86 , srv1040572698:629 } 2024-02-04 08:53:11,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-04 08:53:11,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv622181267=82, srv1491111121=29, srv2104779090=63, srv1072223560=7, srv109882336=8, srv1319347438=19, srv1005020829=0, srv629492384=83, srv1893039794=58, srv1131200254=11, srv1530286860=33, srv1590904846=37, srv813170652=92, srv1554482905=36, srv1059984574=5, srv1233164660=12, srv1500059855=31, srv1498894790=30, srv1553534104=35, srv1964262505=62, srv959558902=99, srv722405515=87, srv128055670=15, srv1867617233=57, srv1782211823=50, srv1396362142=26, srv1730486610=43, srv756278990=89, srv1505310245=32, srv271153800=69, srv424191789=77, srv332021147=73, srv680148768=85, srv346292121=74, srv1600189918=38, srv95953121=98, srv1373084045=24, srv687392325=86, srv23822659=67, srv786095946=90, srv1770073500=49, srv1040572698=3, srv1029616948=1, srv42114356=76, srv644323326=84, srv241630345=68, srv178935620=51, srv1817742175=53, srv51357780=79, srv1603041418=39, srv1757522971=46, srv1961749353=61, srv1299484139=17, srv899263376=97, srv271223098=70, srv1760826513=47, srv1035260777=2, srv1407502069=27, srv1062434145=6, srv295731606=71, srv1925358928=59, srv802773292=91, srv1130941437=10, srv164829939=40, srv1853961011=56, srv1801342508=52, srv892910456=96, srv1320674882=20, srv870910835=95, srv1380065610=25, srv1324199713=22, srv1489666929=28, srv1272447793=14, srv1318963952=18, srv1046129279=4, srv74580547=88, srv1128817224=9, srv472470690=78, srv1285953640=16, srv1322944021=21, srv1685605944=42, srv1551264063=34, srv2125069462=66, srv570622890=81, srv859397182=94, srv567641530=80, srv2114135568=65, srv1839282699=55, srv1945499319=60, srv1764244912=48, srv1742085343=44, srv1755624829=45, srv856933879=93, srv166816278=41, srv1837021797=54, srv32648829=72, srv351529461=75, srv1253531359=13, srv2110409661=64, srv1343517547=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-04 08:53:11,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-04 08:53:11,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-04 08:53:11,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-04 08:53:11,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-04 08:53:11,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-04 08:53:11,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-04 08:53:11,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-04 08:53:11,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-04 08:53:11,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-04 08:53:11,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-04 08:53:11,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-04 08:53:11,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-04 08:53:11,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-04 08:53:11,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-04 08:53:11,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-04 08:53:11,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-04 08:53:11,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-04 08:53:11,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-04 08:53:11,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-04 08:53:11,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-04 08:53:11,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-04 08:53:11,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-04 08:53:11,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-04 08:53:11,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-04 08:53:11,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-04 08:53:11,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-04 08:53:11,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-04 08:53:11,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-04 08:53:11,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-04 08:53:11,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-04 08:53:11,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-04 08:53:11,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-04 08:53:11,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-04 08:53:11,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-04 08:53:11,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-04 08:53:11,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-04 08:53:11,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-04 08:53:11,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-04 08:53:11,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-04 08:53:11,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-04 08:53:11,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-04 08:53:11,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-04 08:53:11,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-04 08:53:11,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-04 08:53:11,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-04 08:53:11,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-04 08:53:11,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-04 08:53:11,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-04 08:53:11,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-04 08:53:11,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-04 08:53:11,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-04 08:53:11,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-04 08:53:11,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-04 08:53:11,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-04 08:53:11,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-04 08:53:11,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-04 08:53:11,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-04 08:53:11,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-04 08:53:11,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-04 08:53:11,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-04 08:53:11,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-04 08:53:11,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-04 08:53:11,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-04 08:53:11,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-04 08:53:11,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-04 08:53:11,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-04 08:53:11,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-04 08:53:11,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-04 08:53:11,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-04 08:53:11,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-04 08:53:11,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-04 08:53:11,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-04 08:53:11,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-04 08:53:11,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-04 08:53:11,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-04 08:53:11,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-04 08:53:11,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-04 08:53:11,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-04 08:53:11,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-04 08:53:11,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-04 08:53:11,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-04 08:53:11,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-04 08:53:11,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-04 08:53:11,537 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:53:11,537 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-04 08:53:11,537 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.238854136976613E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-04 08:53:11,799 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 367 ms to try 5000 different iterations. Found a solution that moves 34 regions; Going from a computed imbalance of 4.238854136976613E-5 to a new imbalance of 7.6258305887507095E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.003777777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:53:11,800 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1035260777:83 , srv1072223560:83 , srv1685605944:83 , srv1742085343:83 , srv1782211823:83 , srv1945499319:83 , srv472470690:83 , srv680148768:83 , srv1005020829:84 , srv1029616948:84 , srv1046129279:84 , srv1059984574:84 , srv1062434145:84 , srv109882336:84 , srv1128817224:84 , srv1130941437:84 , srv1131200254:84 , srv1233164660:84 , srv1253531359:84 , srv1272447793:84 , srv128055670:84 , srv1285953640:84 , srv1299484139:84 , srv1318963952:84 , srv1319347438:84 , srv1320674882:84 , srv1322944021:84 , srv1324199713:84 , srv1343517547:84 , srv1373084045:84 , srv1380065610:84 , srv1396362142:84 , srv1407502069:84 , srv1489666929:84 , srv1491111121:84 , srv1498894790:84 , srv1500059855:84 , srv1505310245:84 , srv1530286860:84 , srv1551264063:84 , srv1554482905:84 , srv1590904846:84 , srv1600189918:84 , srv1603041418:84 , srv164829939:84 , srv1730486610:84 , srv1755624829:84 , srv1757522971:84 , srv1760826513:84 , srv1764244912:84 , srv1770073500:84 , srv178935620:84 , srv1817742175:84 , srv1837021797:84 , srv1853961011:84 , srv1867617233:84 , srv1893039794:84 , srv1925358928:84 , srv1961749353:84 , srv1964262505:84 , srv2104779090:84 , srv2110409661:84 , srv2114135568:84 , srv2125069462:84 , srv23822659:84 , srv241630345:84 , srv271153800:84 , srv32648829:84 , srv332021147:84 , srv346292121:84 , srv351529461:84 , srv42114356:84 , srv424191789:84 , srv567641530:84 , srv570622890:84 , srv622181267:84 , srv629492384:84 , srv644323326:84 , srv687392325:84 , srv74580547:84 , srv756278990:84 , srv786095946:84 , srv802773292:84 , srv813170652:84 , srv856933879:84 , srv859397182:84 , srv870910835:84 , srv892910456:84 , srv899263376:84 , srv95953121:84 , srv959558902:84 , srv1553534104:85 , srv166816278:85 , srv1801342508:85 , srv1839282699:85 , srv271223098:85 , srv295731606:85 , srv51357780:85 , srv722405515:85 , srv1040572698:684 } 2024-02-04 08:53:11,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-04 08:53:11,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv622181267=82, srv1491111121=29, srv2104779090=63, srv1072223560=7, srv109882336=8, srv1319347438=19, srv1005020829=0, srv629492384=83, srv1893039794=58, srv1131200254=11, srv1530286860=33, srv1590904846=37, srv813170652=92, srv1554482905=36, srv1059984574=5, srv1233164660=12, srv1500059855=31, srv1498894790=30, srv1553534104=35, srv1964262505=62, srv959558902=99, srv722405515=87, srv128055670=15, srv1867617233=57, srv1782211823=50, srv1396362142=26, srv1730486610=43, srv756278990=89, srv1505310245=32, srv271153800=69, srv424191789=77, srv332021147=73, srv680148768=85, srv346292121=74, srv1600189918=38, srv95953121=98, srv1373084045=24, srv687392325=86, srv23822659=67, srv786095946=90, srv1770073500=49, srv1040572698=3, srv1029616948=1, srv42114356=76, srv644323326=84, srv241630345=68, srv178935620=51, srv1817742175=53, srv51357780=79, srv1603041418=39, srv1757522971=46, srv1961749353=61, srv1299484139=17, srv899263376=97, srv271223098=70, srv1760826513=47, srv1035260777=2, srv1407502069=27, srv1062434145=6, srv295731606=71, srv1925358928=59, srv802773292=91, srv1130941437=10, srv164829939=40, srv1853961011=56, srv1801342508=52, srv892910456=96, srv1320674882=20, srv870910835=95, srv1380065610=25, srv1324199713=22, srv1489666929=28, srv1272447793=14, srv1318963952=18, srv1046129279=4, srv74580547=88, srv1128817224=9, srv472470690=78, srv1285953640=16, srv1322944021=21, srv1685605944=42, srv1551264063=34, srv2125069462=66, srv570622890=81, srv859397182=94, srv567641530=80, srv2114135568=65, srv1839282699=55, srv1945499319=60, srv1764244912=48, srv1742085343=44, srv1755624829=45, srv856933879=93, srv166816278=41, srv1837021797=54, srv32648829=72, srv351529461=75, srv1253531359=13, srv2110409661=64, srv1343517547=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-04 08:53:11,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-04 08:53:11,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-04 08:53:11,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-04 08:53:11,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-04 08:53:11,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-04 08:53:11,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-04 08:53:11,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-04 08:53:11,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-04 08:53:11,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-04 08:53:11,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-04 08:53:11,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-04 08:53:11,938 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:53:11,938 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1035260777:83 , srv1072223560:83 , srv1685605944:83 , srv1742085343:83 , srv1782211823:83 , srv1945499319:83 , srv472470690:83 , srv680148768:83 , srv1005020829:84 , srv1029616948:84 , srv1046129279:84 , srv1059984574:84 , srv1062434145:84 , srv109882336:84 , srv1128817224:84 , srv1130941437:84 , srv1131200254:84 , srv1233164660:84 , srv1253531359:84 , srv1272447793:84 , srv128055670:84 , srv1285953640:84 , srv1299484139:84 , srv1318963952:84 , srv1319347438:84 , srv1320674882:84 , srv1322944021:84 , srv1324199713:84 , srv1343517547:84 , srv1373084045:84 , srv1380065610:84 , srv1396362142:84 , srv1407502069:84 , srv1489666929:84 , srv1491111121:84 , srv1498894790:84 , srv1500059855:84 , srv1505310245:84 , srv1530286860:84 , srv1551264063:84 , srv1554482905:84 , srv1590904846:84 , srv1600189918:84 , srv1603041418:84 , srv164829939:84 , srv1730486610:84 , srv1755624829:84 , srv1757522971:84 , srv1760826513:84 , srv1764244912:84 , srv1770073500:84 , srv178935620:84 , srv1817742175:84 , srv1837021797:84 , srv1853961011:84 , srv1867617233:84 , srv1893039794:84 , srv1925358928:84 , srv1961749353:84 , srv1964262505:84 , srv2104779090:84 , srv2110409661:84 , srv2114135568:84 , srv2125069462:84 , srv23822659:84 , srv241630345:84 , srv271153800:84 , srv32648829:84 , srv332021147:84 , srv346292121:84 , srv351529461:84 , srv42114356:84 , srv424191789:84 , srv567641530:84 , srv570622890:84 , srv622181267:84 , srv629492384:84 , srv644323326:84 , srv687392325:84 , srv74580547:84 , srv756278990:84 , srv786095946:84 , srv802773292:84 , srv813170652:84 , srv856933879:84 , srv859397182:84 , srv870910835:84 , srv892910456:84 , srv899263376:84 , srv95953121:84 , srv959558902:84 , srv1553534104:85 , srv166816278:85 , srv1801342508:85 , srv1839282699:85 , srv271223098:85 , srv295731606:85 , srv51357780:85 , srv722405515:85 , srv1040572698:684 } 2024-02-04 08:53:11,974 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=370 (was 297) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=2096 (was 3037) 2024-02-04 08:53:11,985 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=370, ProcessCount=167, AvailableMemoryMB=2096 2024-02-04 08:53:11,986 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-04 08:53:11,986 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-02-04 08:53:12,005 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1082591114:84 , srv1207760524:84 , srv1220133808:84 , srv1224770543:84 , srv1263495844:84 , srv1300347796:84 , srv1473021007:84 , srv1483867217:84 , srv1502138720:84 , srv1567782873:84 , srv1687076608:84 , srv1702286391:84 , srv1773487441:84 , srv1807594138:84 , srv1840622359:84 , srv1854467617:84 , srv1903881670:84 , srv1914445796:84 , srv1938189929:84 , srv1942483301:84 , srv2096922942:84 , srv492009363:84 , srv52135229:84 , srv562594506:84 , srv65086467:84 , srv725681651:84 , srv771091169:84 , srv854693137:84 , srv936775772:84 , srv1817325849:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-02-04 08:53:12,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-04 08:53:12,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv936775772=29, srv65086467=25, srv1300347796=5, srv1567782873=9, srv562594506=24, srv1942483301=20, srv1473021007=6, srv1687076608=10, srv1483867217=7, srv52135229=23, srv725681651=26, srv1817325849=14, srv1773487441=12, srv1220133808=2, srv1854467617=16, srv1082591114=0, srv1263495844=4, srv1903881670=17, srv1938189929=19, srv771091169=27, srv1207760524=1, srv492009363=22, srv1502138720=8, srv2096922942=21, srv1702286391=11, srv1807594138=13, srv1840622359=15, srv1224770543=3, srv1914445796=18, srv854693137=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-04 08:53:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-04 08:53:12,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-04 08:53:12,040 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.071059319587977, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:53:12,041 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912416948860002, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.071059319587977, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-02-04 08:55:12,022 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6165907 different iterations. Found a solution that moves 1853 regions; Going from a computed imbalance of 0.9912416948860002 to a new imbalance of 4.324994439959734E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6862962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:55:12,030 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1082591114:90 , srv1207760524:90 , srv1220133808:90 , srv1224770543:90 , srv1263495844:90 , srv1300347796:90 , srv1473021007:90 , srv1483867217:90 , srv1502138720:90 , srv1567782873:90 , srv1687076608:90 , srv1702286391:90 , srv1773487441:90 , srv1807594138:90 , srv1817325849:90 , srv1840622359:90 , srv1854467617:90 , srv1903881670:90 , srv1914445796:90 , srv1938189929:90 , srv1942483301:90 , srv2096922942:90 , srv492009363:90 , srv52135229:90 , srv562594506:90 , srv65086467:90 , srv725681651:90 , srv771091169:90 , srv854693137:90 , srv936775772:90 } 2024-02-04 08:55:12,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-04 08:55:12,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv936775772=29, srv65086467=25, srv1300347796=5, srv1567782873=9, srv562594506=24, srv1942483301=20, srv1473021007=6, srv1687076608=10, srv1483867217=7, srv52135229=23, srv725681651=26, srv1817325849=14, srv1773487441=12, srv1220133808=2, srv1854467617=16, srv1082591114=0, srv1263495844=4, srv1903881670=17, srv1938189929=19, srv771091169=27, srv1207760524=1, srv492009363=22, srv1502138720=8, srv2096922942=21, srv1702286391=11, srv1807594138=13, srv1840622359=15, srv1224770543=3, srv1914445796=18, srv854693137=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-04 08:55:12,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-04 08:55:12,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-04 08:55:12,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-04 08:55:12,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-04 08:55:12,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-04 08:55:12,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-04 08:55:12,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-04 08:55:12,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-04 08:55:12,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-04 08:55:12,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-04 08:55:12,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-04 08:55:12,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-04 08:55:12,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-04 08:55:12,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-04 08:55:12,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-04 08:55:12,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-04 08:55:12,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-04 08:55:12,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-04 08:55:12,056 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-04 08:55:12,056 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1082591114:90 , srv1207760524:90 , srv1220133808:90 , srv1224770543:90 , srv1263495844:90 , srv1300347796:90 , srv1473021007:90 , srv1483867217:90 , srv1502138720:90 , srv1567782873:90 , srv1687076608:90 , srv1702286391:90 , srv1773487441:90 , srv1807594138:90 , srv1817325849:90 , srv1840622359:90 , srv1854467617:90 , srv1903881670:90 , srv1914445796:90 , srv1938189929:90 , srv1942483301:90 , srv2096922942:90 , srv492009363:90 , srv52135229:90 , srv562594506:90 , srv65086467:90 , srv725681651:90 , srv771091169:90 , srv854693137:90 , srv936775772:90 } 2024-02-04 08:55:12,070 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=211 (was 370), ProcessCount=167 (was 167), AvailableMemoryMB=2191 (was 2096) - AvailableMemoryMB LEAK? -