2023-04-16 20:55:35,973 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-04-16 20:55:36,377 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 20:55:36,526 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-16 20:55:36,589 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-16 20:55:36,597 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-16 20:55:36,611 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=377, ProcessCount=169, AvailableMemoryMB=1369 2023-04-16 20:55:36,613 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-16 20:55:36,614 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-16 20:55:36,769 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1037235865:84 , srv1042236700:84 , srv1051853781:84 , srv1074741805:84 , srv1082593712:84 , srv1138363837:84 , srv1160522245:84 , srv1200891904:84 , srv1205940391:84 , srv1206591083:84 , srv1244719063:84 , srv1266563183:84 , srv1277542329:84 , srv1297808936:84 , srv1313404855:84 , srv1315535517:84 , srv133905646:84 , srv1350470401:84 , srv1352208182:84 , srv1360418012:84 , srv1403429916:84 , srv1410056569:84 , srv1421461451:84 , srv1486113043:84 , srv1516933858:84 , srv1521761212:84 , srv1529728482:84 , srv1543560953:84 , srv1549104828:84 , srv1566355633:84 , srv1569689819:84 , srv1580243305:84 , srv1593505740:84 , srv1610042082:84 , srv1625511378:84 , srv1721762127:84 , srv1735415896:84 , srv1754792453:84 , srv1762944501:84 , srv1784674327:84 , srv1817773980:84 , srv1894625276:84 , srv1917239806:84 , srv1937063805:84 , srv1944341072:84 , srv1968618711:84 , srv1974734972:84 , srv1978850928:84 , srv1982150796:84 , srv2023725106:84 , srv2042434109:84 , srv2050098753:84 , srv2089229867:84 , srv2121487251:84 , srv2137307654:84 , srv2141720134:84 , srv225959036:84 , srv231997951:84 , srv296255991:84 , srv303142508:84 , srv329947076:84 , srv345936083:84 , srv36418608:84 , srv379964049:84 , srv41543449:84 , srv480543460:84 , srv50914240:84 , srv542022774:84 , srv572333656:84 , srv582466234:84 , srv59093425:84 , srv617861461:84 , srv620660338:84 , srv666208248:84 , srv668465979:84 , srv684371613:84 , srv717298218:84 , srv747877195:84 , srv752438543:84 , srv753047405:84 , srv757550758:84 , srv763662732:84 , srv764762110:84 , srv790607647:84 , srv792463969:84 , srv796505667:84 , srv805850394:84 , srv824744380:84 , srv844477396:84 , srv85760024:84 , srv873384531:84 , srv891996884:84 , srv893544389:84 , srv915976715:84 , srv950082280:84 , srv953958555:84 , srv961281890:84 , srv970258704:84 , srv979758353:84 , srv192635024:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-04-16 20:55:36,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 20:55:36,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1074741805=3, srv85760024=90, srv979758353=99, srv1486113043=23, srv379964049=64, srv1410056569=21, srv1350470401=17, srv1937063805=44, srv620660338=73, srv192635024=43, srv1735415896=36, srv296255991=59, srv1160522245=6, srv666208248=74, srv1313404855=14, srv542022774=68, srv1762944501=38, srv950082280=95, srv1521761212=25, srv1817773980=40, srv50914240=67, srv1244719063=10, srv480543460=66, srv790607647=84, srv953958555=96, srv764762110=83, srv1529728482=26, srv225959036=57, srv893544389=93, srv1968618711=46, srv2050098753=52, srv844477396=89, srv1051853781=2, srv1266563183=11, srv747877195=78, srv1917239806=42, srv752438543=79, srv41543449=65, srv2089229867=53, srv231997951=58, srv2042434109=51, srv1754792453=37, srv1593505740=32, srv1721762127=35, srv753047405=80, srv873384531=91, srv36418608=63, srv757550758=81, srv2121487251=54, srv1205940391=8, srv717298218=77, srv1894625276=41, srv1549104828=28, srv582466234=70, srv763662732=82, srv1566355633=29, srv59093425=71, srv1543560953=27, srv1569689819=30, srv1277542329=12, srv345936083=62, srv1206591083=9, srv1352208182=18, srv2137307654=55, srv684371613=76, srv1042236700=1, srv1297808936=13, srv133905646=16, srv961281890=97, srv805850394=87, srv572333656=69, srv796505667=86, srv329947076=61, srv1138363837=5, srv1580243305=31, srv891996884=92, srv792463969=85, srv1625511378=34, srv303142508=60, srv1315535517=15, srv1037235865=0, srv1516933858=24, srv1082593712=4, srv1944341072=45, srv970258704=98, srv2141720134=56, srv1403429916=20, srv1982150796=49, srv2023725106=50, srv1974734972=47, srv1360418012=19, srv915976715=94, srv1784674327=39, srv668465979=75, srv824744380=88, srv1421461451=22, srv1610042082=33, srv1978850928=48, srv1200891904=7, srv617861461=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 20:55:36,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 20:55:36,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 20:55:36,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 20:55:36,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 20:55:36,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 20:55:36,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 20:55:36,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 20:55:36,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 20:55:36,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 20:55:36,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 20:55:36,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 20:55:36,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 20:55:36,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 20:55:36,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 20:55:36,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 20:55:36,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 20:55:36,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 20:55:36,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 20:55:36,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 20:55:36,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 20:55:36,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 20:55:36,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 20:55:36,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 20:55:36,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 20:55:36,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 20:55:36,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 20:55:36,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 20:55:36,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 20:55:36,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 20:55:36,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 20:55:36,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-16 20:55:36,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-16 20:55:36,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-16 20:55:36,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-16 20:55:36,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-16 20:55:36,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-16 20:55:36,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-16 20:55:36,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-16 20:55:36,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-16 20:55:36,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-16 20:55:36,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-16 20:55:36,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-16 20:55:36,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-16 20:55:36,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-16 20:55:36,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-16 20:55:36,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-16 20:55:36,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-16 20:55:36,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-16 20:55:36,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-16 20:55:36,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-16 20:55:36,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-16 20:55:36,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-16 20:55:36,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-16 20:55:36,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-16 20:55:36,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-16 20:55:36,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-16 20:55:36,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-16 20:55:36,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-16 20:55:36,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-16 20:55:36,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-16 20:55:36,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-16 20:55:36,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-16 20:55:36,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-16 20:55:36,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-16 20:55:36,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-16 20:55:36,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-16 20:55:36,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-16 20:55:36,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-16 20:55:36,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-16 20:55:36,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-16 20:55:36,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-16 20:55:36,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-16 20:55:36,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-16 20:55:36,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-16 20:55:36,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-16 20:55:36,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-16 20:55:36,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-16 20:55:36,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-16 20:55:36,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-16 20:55:36,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-16 20:55:36,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-16 20:55:36,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-16 20:55:36,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-16 20:55:36,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-16 20:55:36,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-16 20:55:36,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-16 20:55:36,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-16 20:55:36,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-16 20:55:36,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-16 20:55:36,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-16 20:55:36,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-16 20:55:36,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-16 20:55:36,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-16 20:55:36,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-16 20:55:36,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-16 20:55:36,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-16 20:55:36,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-16 20:55:36,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-16 20:55:36,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-16 20:55:36,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-16 20:55:36,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-16 20:55:37,201 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:55:37,202 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-16 20:55:37,202 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-16 20:55:37,580 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 596 ms to try 5000 different iterations. Found a solution that moves 4692 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1777845258745694. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04309743265941889, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060463974860178155, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5213333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04309743265941889, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17325, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23658333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:55:37,614 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv133905646:81 , srv1549104828:81 , srv1894625276:81 , srv1313404855:82 , srv1566355633:82 , srv790607647:82 , srv1817773980:83 , srv1937063805:83 , srv1978850928:83 , srv2050098753:83 , srv41543449:83 , srv480543460:83 , srv666208248:83 , srv792463969:83 , srv85760024:83 , srv1037235865:84 , srv1082593712:84 , srv1206591083:84 , srv1410056569:84 , srv1421461451:84 , srv1784674327:84 , srv1968618711:84 , srv225959036:84 , srv231997951:84 , srv542022774:84 , srv717298218:84 , srv953958555:84 , srv1042236700:85 , srv1051853781:85 , srv1244719063:85 , srv1297808936:85 , srv1352208182:85 , srv1360418012:85 , srv1403429916:85 , srv1516933858:85 , srv1529728482:85 , srv2042434109:85 , srv2121487251:85 , srv2141720134:85 , srv303142508:85 , srv329947076:85 , srv50914240:85 , srv572333656:85 , srv747877195:85 , srv753047405:85 , srv764762110:85 , srv805850394:85 , srv891996884:85 , srv970258704:85 , srv1074741805:86 , srv1138363837:86 , srv1200891904:86 , srv1593505740:86 , srv1974734972:86 , srv345936083:86 , srv36418608:86 , srv617861461:86 , srv757550758:86 , srv763662732:86 , srv873384531:86 , srv1205940391:87 , srv1266563183:87 , srv1610042082:87 , srv1625511378:87 , srv1917239806:87 , srv1944341072:87 , srv2089229867:87 , srv296255991:87 , srv668465979:87 , srv684371613:87 , srv824744380:87 , srv915976715:87 , srv979758353:87 , srv1277542329:88 , srv1350470401:88 , srv1569689819:88 , srv1721762127:88 , srv1982150796:88 , srv796505667:88 , srv844477396:88 , srv893544389:88 , srv1486113043:89 , srv1521761212:89 , srv1580243305:89 , srv2023725106:89 , srv2137307654:89 , srv752438543:89 , srv950082280:89 , srv961281890:89 , srv1160522245:91 , srv1735415896:91 , srv1754792453:91 , srv379964049:91 , srv582466234:91 , srv59093425:91 , srv620660338:91 , srv1315535517:94 , srv1543560953:94 , srv1762944501:95 , srv192635024:473 } 2023-04-16 20:55:37,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 20:55:37,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1074741805=3, srv85760024=90, srv979758353=99, srv1486113043=23, srv379964049=64, srv1410056569=21, srv1350470401=17, srv1937063805=44, srv620660338=73, srv192635024=43, srv1735415896=36, srv296255991=59, srv1160522245=6, srv666208248=74, srv1313404855=14, srv542022774=68, srv1762944501=38, srv950082280=95, srv1521761212=25, srv1817773980=40, srv50914240=67, srv1244719063=10, srv480543460=66, srv790607647=84, srv953958555=96, srv764762110=83, srv1529728482=26, srv225959036=57, srv893544389=93, srv1968618711=46, srv2050098753=52, srv844477396=89, srv1051853781=2, srv1266563183=11, srv747877195=78, srv1917239806=42, srv752438543=79, srv41543449=65, srv2089229867=53, srv231997951=58, srv2042434109=51, srv1754792453=37, srv1593505740=32, srv1721762127=35, srv753047405=80, srv873384531=91, srv36418608=63, srv757550758=81, srv2121487251=54, srv1205940391=8, srv717298218=77, srv1894625276=41, srv1549104828=28, srv582466234=70, srv763662732=82, srv1566355633=29, srv59093425=71, srv1543560953=27, srv1569689819=30, srv1277542329=12, srv345936083=62, srv1206591083=9, srv1352208182=18, srv2137307654=55, srv684371613=76, srv1042236700=1, srv1297808936=13, srv133905646=16, srv961281890=97, srv805850394=87, srv572333656=69, srv796505667=86, srv329947076=61, srv1138363837=5, srv1580243305=31, srv891996884=92, srv792463969=85, srv1625511378=34, srv303142508=60, srv1315535517=15, srv1037235865=0, srv1516933858=24, srv1082593712=4, srv1944341072=45, srv970258704=98, srv2141720134=56, srv1403429916=20, srv1982150796=49, srv2023725106=50, srv1974734972=47, srv1360418012=19, srv915976715=94, srv1784674327=39, srv668465979=75, srv824744380=88, srv1421461451=22, srv1610042082=33, srv1978850928=48, srv1200891904=7, srv617861461=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 20:55:37,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 20:55:37,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 20:55:37,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 20:55:37,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 20:55:37,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 20:55:37,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 20:55:37,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 20:55:37,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 20:55:37,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 20:55:37,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 20:55:37,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 20:55:37,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 20:55:37,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 20:55:37,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 20:55:37,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 20:55:37,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 20:55:37,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 20:55:37,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 20:55:37,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 20:55:37,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 20:55:37,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 20:55:37,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 20:55:37,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 20:55:37,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 20:55:37,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 20:55:37,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 20:55:37,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 20:55:37,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 20:55:37,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 20:55:37,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 20:55:37,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-16 20:55:37,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-16 20:55:37,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-16 20:55:37,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-16 20:55:37,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-16 20:55:37,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-16 20:55:37,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-16 20:55:37,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-16 20:55:37,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-16 20:55:37,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-16 20:55:37,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-16 20:55:37,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-16 20:55:37,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-16 20:55:37,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-16 20:55:37,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-16 20:55:37,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-16 20:55:37,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-16 20:55:37,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-16 20:55:37,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-16 20:55:37,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-16 20:55:37,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-16 20:55:37,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-16 20:55:37,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-16 20:55:37,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-16 20:55:37,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-16 20:55:37,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-16 20:55:37,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-16 20:55:37,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-16 20:55:37,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-16 20:55:37,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-16 20:55:37,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-16 20:55:37,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-16 20:55:37,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-16 20:55:37,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-16 20:55:37,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-16 20:55:37,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-16 20:55:37,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-16 20:55:37,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-16 20:55:37,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-16 20:55:37,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-16 20:55:37,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-16 20:55:37,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-16 20:55:37,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-16 20:55:37,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-16 20:55:37,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-16 20:55:37,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-16 20:55:37,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-16 20:55:37,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-16 20:55:37,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-16 20:55:37,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-16 20:55:37,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-16 20:55:37,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-16 20:55:37,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-16 20:55:37,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-16 20:55:37,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-16 20:55:37,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-16 20:55:37,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-16 20:55:37,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-16 20:55:37,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-16 20:55:37,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-16 20:55:37,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-16 20:55:37,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-16 20:55:37,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-16 20:55:37,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-16 20:55:37,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-16 20:55:37,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-16 20:55:37,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-16 20:55:37,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-16 20:55:37,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-16 20:55:37,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-16 20:55:37,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-16 20:55:37,822 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04309743265941889, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060463974860178155, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04309743265941889, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17325, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23658333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:55:37,823 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-16 20:55:37,823 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17775167178836493, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04309743265941889, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060463974860178155, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04309743265941889, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17325, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23658333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-16 20:55:38,221 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 533 ms to try 5000 different iterations. Found a solution that moves 2596 regions; Going from a computed imbalance of 0.17775167178836493 to a new imbalance of 0.013820675857096497. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023734197557164303); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03921794408236028, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28844444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023734197557164303); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0105); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045083333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:55:38,235 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1762944501:75 , srv1315535517:78 , srv1754792453:79 , srv379964049:79 , srv1543560953:80 , srv1735415896:80 , srv582466234:80 , srv1160522245:82 , srv1486113043:82 , srv1721762127:82 , srv59093425:82 , srv620660338:82 , srv752438543:82 , srv1521761212:83 , srv1569689819:83 , srv1580243305:83 , srv1917239806:83 , srv950082280:83 , srv961281890:83 , srv1074741805:84 , srv1205940391:84 , srv1277542329:84 , srv1352208182:84 , srv1625511378:84 , srv1974734972:84 , srv2023725106:84 , srv2089229867:84 , srv2137307654:84 , srv345936083:84 , srv684371613:84 , srv763662732:84 , srv844477396:84 , srv1266563183:85 , srv1529728482:85 , srv2042434109:85 , srv668465979:85 , srv747877195:85 , srv873384531:85 , srv915976715:85 , srv979758353:85 , srv1297808936:86 , srv1968618711:86 , srv2121487251:86 , srv231997951:86 , srv296255991:86 , srv36418608:86 , srv50914240:86 , srv572333656:86 , srv717298218:86 , srv796505667:86 , srv805850394:86 , srv824744380:86 , srv891996884:86 , srv970258704:86 , srv1051853781:87 , srv1082593712:87 , srv1138363837:87 , srv1200891904:87 , srv1350470401:87 , srv1403429916:87 , srv1516933858:87 , srv1610042082:87 , srv1982150796:87 , srv2141720134:87 , srv225959036:87 , srv303142508:87 , srv329947076:87 , srv617861461:87 , srv666208248:87 , srv757550758:87 , srv764762110:87 , srv953958555:87 , srv1037235865:88 , srv1042236700:88 , srv1244719063:88 , srv1410056569:88 , srv1421461451:88 , srv1593505740:88 , srv1944341072:88 , srv41543449:88 , srv480543460:88 , srv542022774:88 , srv753047405:88 , srv792463969:88 , srv85760024:88 , srv893544389:88 , srv1206591083:89 , srv1360418012:89 , srv1549104828:89 , srv1784674327:89 , srv1937063805:89 , srv1978850928:89 , srv1313404855:90 , srv790607647:90 , srv1566355633:91 , srv2050098753:91 , srv133905646:92 , srv1894625276:92 , srv1817773980:93 , srv192635024:512 } 2023-04-16 20:55:38,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 20:55:38,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1074741805=3, srv85760024=90, srv979758353=99, srv1486113043=23, srv379964049=64, srv1410056569=21, srv1350470401=17, srv1937063805=44, srv620660338=73, srv192635024=43, srv1735415896=36, srv296255991=59, srv1160522245=6, srv666208248=74, srv1313404855=14, srv542022774=68, srv1762944501=38, srv950082280=95, srv1521761212=25, srv1817773980=40, srv50914240=67, srv1244719063=10, srv480543460=66, srv790607647=84, srv953958555=96, srv764762110=83, srv1529728482=26, srv225959036=57, srv893544389=93, srv1968618711=46, srv2050098753=52, srv844477396=89, srv1051853781=2, srv1266563183=11, srv747877195=78, srv1917239806=42, srv752438543=79, srv41543449=65, srv2089229867=53, srv231997951=58, srv2042434109=51, srv1754792453=37, srv1593505740=32, srv1721762127=35, srv753047405=80, srv873384531=91, srv36418608=63, srv757550758=81, srv2121487251=54, srv1205940391=8, srv717298218=77, srv1894625276=41, srv1549104828=28, srv582466234=70, srv763662732=82, srv1566355633=29, srv59093425=71, srv1543560953=27, srv1569689819=30, srv1277542329=12, srv345936083=62, srv1206591083=9, srv1352208182=18, srv2137307654=55, srv684371613=76, srv1042236700=1, srv1297808936=13, srv133905646=16, srv961281890=97, srv805850394=87, srv572333656=69, srv796505667=86, srv329947076=61, srv1138363837=5, srv1580243305=31, srv891996884=92, srv792463969=85, srv1625511378=34, srv303142508=60, srv1315535517=15, srv1037235865=0, srv1516933858=24, srv1082593712=4, srv1944341072=45, srv970258704=98, srv2141720134=56, srv1403429916=20, srv1982150796=49, srv2023725106=50, srv1974734972=47, srv1360418012=19, srv915976715=94, srv1784674327=39, srv668465979=75, srv824744380=88, srv1421461451=22, srv1610042082=33, srv1978850928=48, srv1200891904=7, srv617861461=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 20:55:38,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 20:55:38,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 20:55:38,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 20:55:38,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 20:55:38,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 20:55:38,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 20:55:38,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 20:55:38,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 20:55:38,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 20:55:38,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 20:55:38,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 20:55:38,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 20:55:38,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 20:55:38,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 20:55:38,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 20:55:38,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 20:55:38,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 20:55:38,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 20:55:38,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 20:55:38,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 20:55:38,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 20:55:38,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 20:55:38,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 20:55:38,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 20:55:38,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 20:55:38,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 20:55:38,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 20:55:38,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 20:55:38,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 20:55:38,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 20:55:38,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-16 20:55:38,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-16 20:55:38,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-16 20:55:38,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-16 20:55:38,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-16 20:55:38,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-16 20:55:38,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-16 20:55:38,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-16 20:55:38,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-16 20:55:38,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-16 20:55:38,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-16 20:55:38,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-16 20:55:38,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-16 20:55:38,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-16 20:55:38,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-16 20:55:38,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-16 20:55:38,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-16 20:55:38,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-16 20:55:38,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-16 20:55:38,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-16 20:55:38,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-16 20:55:38,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-16 20:55:38,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-16 20:55:38,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-16 20:55:38,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-16 20:55:38,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-16 20:55:38,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-16 20:55:38,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-16 20:55:38,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-16 20:55:38,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-16 20:55:38,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-16 20:55:38,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-16 20:55:38,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-16 20:55:38,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-16 20:55:38,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-16 20:55:38,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-16 20:55:38,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-16 20:55:38,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-16 20:55:38,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-16 20:55:38,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-16 20:55:38,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-16 20:55:38,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-16 20:55:38,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-16 20:55:38,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-16 20:55:38,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-16 20:55:38,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-16 20:55:38,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-16 20:55:38,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-16 20:55:38,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-16 20:55:38,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-16 20:55:38,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-16 20:55:38,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-16 20:55:38,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-16 20:55:38,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-16 20:55:38,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-16 20:55:38,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-16 20:55:38,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-16 20:55:38,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-16 20:55:38,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-16 20:55:38,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-16 20:55:38,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-16 20:55:38,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-16 20:55:38,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-16 20:55:38,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-16 20:55:38,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-16 20:55:38,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-16 20:55:38,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-16 20:55:38,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-16 20:55:38,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-16 20:55:38,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-16 20:55:38,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-16 20:55:38,428 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023734197557164303); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03921794408236028, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023734197557164303); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0105); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045083333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:55:38,429 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-16 20:55:38,429 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013802498276579277, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023734197557164303); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03921794408236028, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023734197557164303); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0105); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045083333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-16 20:55:38,858 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 566 ms to try 5000 different iterations. Found a solution that moves 1031 regions; Going from a computed imbalance of 0.013802498276579277 to a new imbalance of 7.336005871910984E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007783387427102616); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008040302522073696); MoveCostFunction : (multiplier=7.0, imbalance=0.11455555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007783387427102616); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:55:38,862 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv893544389:80 , srv1817773980:82 , srv1982150796:82 , srv1138363837:83 , srv1350470401:83 , srv1593505740:83 , srv1610042082:83 , srv1944341072:83 , srv1042236700:84 , srv1200891904:84 , srv1244719063:84 , srv1360418012:84 , srv1543560953:84 , srv1784674327:84 , srv1894625276:84 , srv1978850928:84 , srv2023725106:84 , srv2050098753:84 , srv2137307654:84 , srv303142508:84 , srv59093425:84 , srv617861461:84 , srv620660338:84 , srv757550758:84 , srv790607647:84 , srv796505667:84 , srv979758353:84 , srv1082593712:85 , srv1160522245:85 , srv1297808936:85 , srv1315535517:85 , srv133905646:85 , srv1521761212:85 , srv1566355633:85 , srv2141720134:85 , srv296255991:85 , srv329947076:85 , srv542022774:85 , srv668465979:85 , srv684371613:85 , srv753047405:85 , srv764762110:85 , srv824744380:85 , srv950082280:85 , srv1037235865:86 , srv1051853781:86 , srv1074741805:86 , srv1205940391:86 , srv1206591083:86 , srv1266563183:86 , srv1277542329:86 , srv1313404855:86 , srv1403429916:86 , srv1410056569:86 , srv1421461451:86 , srv1486113043:86 , srv1516933858:86 , srv1569689819:86 , srv1580243305:86 , srv1625511378:86 , srv1735415896:86 , srv1937063805:86 , srv1968618711:86 , srv2121487251:86 , srv231997951:86 , srv36418608:86 , srv41543449:86 , srv572333656:86 , srv582466234:86 , srv752438543:86 , srv792463969:86 , srv805850394:86 , srv844477396:86 , srv873384531:86 , srv891996884:86 , srv915976715:86 , srv961281890:86 , srv970258704:86 , srv1529728482:87 , srv1754792453:87 , srv1974734972:87 , srv2042434109:87 , srv2089229867:87 , srv225959036:87 , srv345936083:87 , srv379964049:87 , srv480543460:87 , srv50914240:87 , srv717298218:87 , srv747877195:87 , srv763662732:87 , srv85760024:87 , srv953958555:87 , srv1352208182:88 , srv1549104828:88 , srv1721762127:88 , srv1762944501:88 , srv1917239806:88 , srv666208248:89 , srv192635024:542 } 2023-04-16 20:55:38,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 20:55:38,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1074741805=3, srv85760024=90, srv979758353=99, srv1486113043=23, srv379964049=64, srv1410056569=21, srv1350470401=17, srv1937063805=44, srv620660338=73, srv192635024=43, srv1735415896=36, srv296255991=59, srv1160522245=6, srv666208248=74, srv1313404855=14, srv542022774=68, srv1762944501=38, srv950082280=95, srv1521761212=25, srv1817773980=40, srv50914240=67, srv1244719063=10, srv480543460=66, srv790607647=84, srv953958555=96, srv764762110=83, srv1529728482=26, srv225959036=57, srv893544389=93, srv1968618711=46, srv2050098753=52, srv844477396=89, srv1051853781=2, srv1266563183=11, srv747877195=78, srv1917239806=42, srv752438543=79, srv41543449=65, srv2089229867=53, srv231997951=58, srv2042434109=51, srv1754792453=37, srv1593505740=32, srv1721762127=35, srv753047405=80, srv873384531=91, srv36418608=63, srv757550758=81, srv2121487251=54, srv1205940391=8, srv717298218=77, srv1894625276=41, srv1549104828=28, srv582466234=70, srv763662732=82, srv1566355633=29, srv59093425=71, srv1543560953=27, srv1569689819=30, srv1277542329=12, srv345936083=62, srv1206591083=9, srv1352208182=18, srv2137307654=55, srv684371613=76, srv1042236700=1, srv1297808936=13, srv133905646=16, srv961281890=97, srv805850394=87, srv572333656=69, srv796505667=86, srv329947076=61, srv1138363837=5, srv1580243305=31, srv891996884=92, srv792463969=85, srv1625511378=34, srv303142508=60, srv1315535517=15, srv1037235865=0, srv1516933858=24, srv1082593712=4, srv1944341072=45, srv970258704=98, srv2141720134=56, srv1403429916=20, srv1982150796=49, srv2023725106=50, srv1974734972=47, srv1360418012=19, srv915976715=94, srv1784674327=39, srv668465979=75, srv824744380=88, srv1421461451=22, srv1610042082=33, srv1978850928=48, srv1200891904=7, srv617861461=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 20:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 20:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 20:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 20:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 20:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 20:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 20:55:38,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 20:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 20:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 20:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 20:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 20:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 20:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 20:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 20:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 20:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 20:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 20:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 20:55:38,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 20:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 20:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 20:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 20:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 20:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 20:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 20:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 20:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 20:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 20:55:38,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 20:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 20:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-16 20:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-16 20:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-16 20:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-16 20:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-16 20:55:38,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-16 20:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-16 20:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-16 20:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-16 20:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-16 20:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-16 20:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-16 20:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-16 20:55:38,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-16 20:55:38,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-16 20:55:38,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-16 20:55:38,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-16 20:55:38,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-16 20:55:38,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-16 20:55:38,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-16 20:55:38,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-16 20:55:38,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-16 20:55:38,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-16 20:55:38,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-16 20:55:38,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-16 20:55:38,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-16 20:55:38,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-16 20:55:38,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-16 20:55:38,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-16 20:55:38,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-16 20:55:38,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-16 20:55:38,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-16 20:55:38,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-16 20:55:38,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-16 20:55:38,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-16 20:55:38,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-16 20:55:38,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-16 20:55:38,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-16 20:55:38,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-16 20:55:38,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-16 20:55:38,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-16 20:55:38,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-16 20:55:38,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-16 20:55:38,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-16 20:55:38,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-16 20:55:38,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-16 20:55:38,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-16 20:55:38,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-16 20:55:38,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-16 20:55:38,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-16 20:55:38,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-16 20:55:38,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-16 20:55:38,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-16 20:55:38,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-16 20:55:38,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-16 20:55:38,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-16 20:55:38,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-16 20:55:38,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-16 20:55:38,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-16 20:55:38,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-16 20:55:38,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-16 20:55:38,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-16 20:55:38,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-16 20:55:38,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-16 20:55:38,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-16 20:55:38,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-16 20:55:38,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-16 20:55:38,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-16 20:55:38,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-16 20:55:38,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-16 20:55:38,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-16 20:55:39,058 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007783387427102616); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008040302522073696); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007783387427102616); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:55:39,058 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-16 20:55:39,059 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.263813708916944E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007783387427102616); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008040302522073696); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007783387427102616); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-16 20:55:39,425 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 499 ms to try 5000 different iterations. Found a solution that moves 275 regions; Going from a computed imbalance of 7.263813708916944E-4 to a new imbalance of 4.310041838962775E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.030555555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:55:39,426 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1206591083:83 , srv1266563183:83 , srv1543560953:83 , srv1721762127:83 , srv296255991:83 , srv666208248:83 , srv1051853781:84 , srv1160522245:84 , srv1200891904:84 , srv1277542329:84 , srv133905646:84 , srv1350470401:84 , srv1360418012:84 , srv1403429916:84 , srv1410056569:84 , srv1421461451:84 , srv1486113043:84 , srv1516933858:84 , srv1549104828:84 , srv1566355633:84 , srv1580243305:84 , srv1762944501:84 , srv1784674327:84 , srv1917239806:84 , srv1937063805:84 , srv1944341072:84 , srv2050098753:84 , srv2089229867:84 , srv36418608:84 , srv480543460:84 , srv50914240:84 , srv617861461:84 , srv753047405:84 , srv805850394:84 , srv824744380:84 , srv844477396:84 , srv85760024:84 , srv873384531:84 , srv915976715:84 , srv953958555:84 , srv961281890:84 , srv1037235865:85 , srv1042236700:85 , srv1138363837:85 , srv1205940391:85 , srv1244719063:85 , srv1313404855:85 , srv1315535517:85 , srv1352208182:85 , srv1521761212:85 , srv1569689819:85 , srv1593505740:85 , srv1610042082:85 , srv1735415896:85 , srv1754792453:85 , srv1817773980:85 , srv1974734972:85 , srv1982150796:85 , srv2042434109:85 , srv2121487251:85 , srv2137307654:85 , srv2141720134:85 , srv225959036:85 , srv329947076:85 , srv345936083:85 , srv379964049:85 , srv41543449:85 , srv542022774:85 , srv572333656:85 , srv582466234:85 , srv59093425:85 , srv620660338:85 , srv668465979:85 , srv717298218:85 , srv747877195:85 , srv752438543:85 , srv757550758:85 , srv763662732:85 , srv764762110:85 , srv792463969:85 , srv796505667:85 , srv891996884:85 , srv970258704:85 , srv979758353:85 , srv1074741805:86 , srv1297808936:86 , srv1529728482:86 , srv1625511378:86 , srv1894625276:86 , srv1968618711:86 , srv1978850928:86 , srv2023725106:86 , srv231997951:86 , srv303142508:86 , srv684371613:86 , srv790607647:86 , srv950082280:86 , srv1082593712:87 , srv893544389:87 , srv192635024:615 } 2023-04-16 20:55:39,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 20:55:39,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1074741805=3, srv85760024=90, srv979758353=99, srv1486113043=23, srv379964049=64, srv1410056569=21, srv1350470401=17, srv1937063805=44, srv620660338=73, srv192635024=43, srv1735415896=36, srv296255991=59, srv1160522245=6, srv666208248=74, srv1313404855=14, srv542022774=68, srv1762944501=38, srv950082280=95, srv1521761212=25, srv1817773980=40, srv50914240=67, srv1244719063=10, srv480543460=66, srv790607647=84, srv953958555=96, srv764762110=83, srv1529728482=26, srv225959036=57, srv893544389=93, srv1968618711=46, srv2050098753=52, srv844477396=89, srv1051853781=2, srv1266563183=11, srv747877195=78, srv1917239806=42, srv752438543=79, srv41543449=65, srv2089229867=53, srv231997951=58, srv2042434109=51, srv1754792453=37, srv1593505740=32, srv1721762127=35, srv753047405=80, srv873384531=91, srv36418608=63, srv757550758=81, srv2121487251=54, srv1205940391=8, srv717298218=77, srv1894625276=41, srv1549104828=28, srv582466234=70, srv763662732=82, srv1566355633=29, srv59093425=71, srv1543560953=27, srv1569689819=30, srv1277542329=12, srv345936083=62, srv1206591083=9, srv1352208182=18, srv2137307654=55, srv684371613=76, srv1042236700=1, srv1297808936=13, srv133905646=16, srv961281890=97, srv805850394=87, srv572333656=69, srv796505667=86, srv329947076=61, srv1138363837=5, srv1580243305=31, srv891996884=92, srv792463969=85, srv1625511378=34, srv303142508=60, srv1315535517=15, srv1037235865=0, srv1516933858=24, srv1082593712=4, srv1944341072=45, srv970258704=98, srv2141720134=56, srv1403429916=20, srv1982150796=49, srv2023725106=50, srv1974734972=47, srv1360418012=19, srv915976715=94, srv1784674327=39, srv668465979=75, srv824744380=88, srv1421461451=22, srv1610042082=33, srv1978850928=48, srv1200891904=7, srv617861461=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 20:55:39,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 20:55:39,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 20:55:39,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 20:55:39,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 20:55:39,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 20:55:39,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 20:55:39,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 20:55:39,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 20:55:39,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 20:55:39,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 20:55:39,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 20:55:39,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 20:55:39,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 20:55:39,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 20:55:39,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 20:55:39,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 20:55:39,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 20:55:39,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 20:55:39,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 20:55:39,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 20:55:39,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 20:55:39,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 20:55:39,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 20:55:39,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 20:55:39,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 20:55:39,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 20:55:39,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 20:55:39,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 20:55:39,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 20:55:39,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 20:55:39,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-16 20:55:39,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-16 20:55:39,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-16 20:55:39,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-16 20:55:39,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-16 20:55:39,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-16 20:55:39,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-16 20:55:39,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-16 20:55:39,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-16 20:55:39,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-16 20:55:39,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-16 20:55:39,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-16 20:55:39,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-16 20:55:39,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-16 20:55:39,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-16 20:55:39,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-16 20:55:39,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-16 20:55:39,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-16 20:55:39,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-16 20:55:39,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-16 20:55:39,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-16 20:55:39,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-16 20:55:39,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-16 20:55:39,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-16 20:55:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-16 20:55:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-16 20:55:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-16 20:55:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-16 20:55:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-16 20:55:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-16 20:55:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-16 20:55:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-16 20:55:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-16 20:55:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-16 20:55:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-16 20:55:39,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-16 20:55:39,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-16 20:55:39,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-16 20:55:39,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-16 20:55:39,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-16 20:55:39,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-16 20:55:39,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-16 20:55:39,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-16 20:55:39,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-16 20:55:39,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-16 20:55:39,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-16 20:55:39,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-16 20:55:39,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-16 20:55:39,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-16 20:55:39,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-16 20:55:39,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-16 20:55:39,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-16 20:55:39,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-16 20:55:39,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-16 20:55:39,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-16 20:55:39,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-16 20:55:39,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-16 20:55:39,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-16 20:55:39,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-16 20:55:39,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-16 20:55:39,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-16 20:55:39,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-16 20:55:39,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-16 20:55:39,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-16 20:55:39,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-16 20:55:39,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-16 20:55:39,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-16 20:55:39,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-16 20:55:39,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-16 20:55:39,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-16 20:55:39,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-16 20:55:39,609 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:55:39,609 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-16 20:55:39,610 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.117482723314271E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-16 20:55:39,942 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 468 ms to try 5000 different iterations. Found a solution that moves 32 regions; Going from a computed imbalance of 4.117482723314271E-5 to a new imbalance of 6.968137842950207E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0035555555555555557); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:55:39,943 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1037235865:83 , srv1082593712:83 , srv1313404855:83 , srv1529728482:83 , srv1625511378:83 , srv1817773980:83 , srv1894625276:83 , srv2023725106:83 , srv225959036:83 , srv796505667:83 , srv893544389:83 , srv950082280:83 , srv1042236700:84 , srv1051853781:84 , srv1074741805:84 , srv1160522245:84 , srv1205940391:84 , srv1206591083:84 , srv1244719063:84 , srv1277542329:84 , srv1297808936:84 , srv1315535517:84 , srv133905646:84 , srv1350470401:84 , srv1352208182:84 , srv1360418012:84 , srv1403429916:84 , srv1410056569:84 , srv1421461451:84 , srv1516933858:84 , srv1521761212:84 , srv1549104828:84 , srv1566355633:84 , srv1569689819:84 , srv1580243305:84 , srv1593505740:84 , srv1610042082:84 , srv1735415896:84 , srv1754792453:84 , srv1762944501:84 , srv1917239806:84 , srv1937063805:84 , srv1944341072:84 , srv1968618711:84 , srv1974734972:84 , srv1978850928:84 , srv1982150796:84 , srv2042434109:84 , srv2050098753:84 , srv2089229867:84 , srv2121487251:84 , srv2137307654:84 , srv2141720134:84 , srv231997951:84 , srv303142508:84 , srv329947076:84 , srv345936083:84 , srv36418608:84 , srv379964049:84 , srv41543449:84 , srv480543460:84 , srv50914240:84 , srv542022774:84 , srv572333656:84 , srv582466234:84 , srv59093425:84 , srv620660338:84 , srv666208248:84 , srv668465979:84 , srv684371613:84 , srv717298218:84 , srv747877195:84 , srv752438543:84 , srv753047405:84 , srv757550758:84 , srv763662732:84 , srv764762110:84 , srv790607647:84 , srv792463969:84 , srv824744380:84 , srv844477396:84 , srv85760024:84 , srv891996884:84 , srv915976715:84 , srv953958555:84 , srv961281890:84 , srv970258704:84 , srv1138363837:85 , srv1200891904:85 , srv1266563183:85 , srv1486113043:85 , srv1543560953:85 , srv1721762127:85 , srv1784674327:85 , srv296255991:85 , srv617861461:85 , srv805850394:85 , srv873384531:85 , srv979758353:85 , srv192635024:684 } 2023-04-16 20:55:39,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 20:55:39,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1074741805=3, srv85760024=90, srv979758353=99, srv1486113043=23, srv379964049=64, srv1410056569=21, srv1350470401=17, srv1937063805=44, srv620660338=73, srv192635024=43, srv1735415896=36, srv296255991=59, srv1160522245=6, srv666208248=74, srv1313404855=14, srv542022774=68, srv1762944501=38, srv950082280=95, srv1521761212=25, srv1817773980=40, srv50914240=67, srv1244719063=10, srv480543460=66, srv790607647=84, srv953958555=96, srv764762110=83, srv1529728482=26, srv225959036=57, srv893544389=93, srv1968618711=46, srv2050098753=52, srv844477396=89, srv1051853781=2, srv1266563183=11, srv747877195=78, srv1917239806=42, srv752438543=79, srv41543449=65, srv2089229867=53, srv231997951=58, srv2042434109=51, srv1754792453=37, srv1593505740=32, srv1721762127=35, srv753047405=80, srv873384531=91, srv36418608=63, srv757550758=81, srv2121487251=54, srv1205940391=8, srv717298218=77, srv1894625276=41, srv1549104828=28, srv582466234=70, srv763662732=82, srv1566355633=29, srv59093425=71, srv1543560953=27, srv1569689819=30, srv1277542329=12, srv345936083=62, srv1206591083=9, srv1352208182=18, srv2137307654=55, srv684371613=76, srv1042236700=1, srv1297808936=13, srv133905646=16, srv961281890=97, srv805850394=87, srv572333656=69, srv796505667=86, srv329947076=61, srv1138363837=5, srv1580243305=31, srv891996884=92, srv792463969=85, srv1625511378=34, srv303142508=60, srv1315535517=15, srv1037235865=0, srv1516933858=24, srv1082593712=4, srv1944341072=45, srv970258704=98, srv2141720134=56, srv1403429916=20, srv1982150796=49, srv2023725106=50, srv1974734972=47, srv1360418012=19, srv915976715=94, srv1784674327=39, srv668465979=75, srv824744380=88, srv1421461451=22, srv1610042082=33, srv1978850928=48, srv1200891904=7, srv617861461=72} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 20:55:39,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 20:55:39,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 20:55:39,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 20:55:39,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 20:55:39,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 20:55:39,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 20:55:39,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 20:55:39,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 20:55:39,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 20:55:39,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 20:55:39,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 20:55:39,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 20:55:39,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 20:55:39,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 20:55:39,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 20:55:39,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 20:55:39,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 20:55:39,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 20:55:39,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 20:55:39,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 20:55:39,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 20:55:39,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 20:55:39,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 20:55:39,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 20:55:39,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 20:55:39,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 20:55:39,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 20:55:39,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 20:55:39,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 20:55:39,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 20:55:39,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-16 20:55:39,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-16 20:55:39,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-16 20:55:39,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-16 20:55:39,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-16 20:55:39,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-16 20:55:39,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-16 20:55:39,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-16 20:55:39,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-16 20:55:39,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-16 20:55:39,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-16 20:55:39,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-16 20:55:39,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-16 20:55:39,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-16 20:55:39,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-16 20:55:39,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-16 20:55:39,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-16 20:55:39,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-16 20:55:39,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-16 20:55:39,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-16 20:55:39,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-16 20:55:39,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-16 20:55:39,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-16 20:55:39,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-16 20:55:39,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-16 20:55:39,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-16 20:55:39,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-16 20:55:39,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-16 20:55:39,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-16 20:55:39,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-16 20:55:39,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-16 20:55:39,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-16 20:55:39,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-16 20:55:39,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-16 20:55:39,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-16 20:55:39,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-16 20:55:39,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-16 20:55:39,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-16 20:55:39,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-16 20:55:39,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-16 20:55:39,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-16 20:55:40,104 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:55:40,104 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1037235865:83 , srv1082593712:83 , srv1313404855:83 , srv1529728482:83 , srv1625511378:83 , srv1817773980:83 , srv1894625276:83 , srv2023725106:83 , srv225959036:83 , srv796505667:83 , srv893544389:83 , srv950082280:83 , srv1042236700:84 , srv1051853781:84 , srv1074741805:84 , srv1160522245:84 , srv1205940391:84 , srv1206591083:84 , srv1244719063:84 , srv1277542329:84 , srv1297808936:84 , srv1315535517:84 , srv133905646:84 , srv1350470401:84 , srv1352208182:84 , srv1360418012:84 , srv1403429916:84 , srv1410056569:84 , srv1421461451:84 , srv1516933858:84 , srv1521761212:84 , srv1549104828:84 , srv1566355633:84 , srv1569689819:84 , srv1580243305:84 , srv1593505740:84 , srv1610042082:84 , srv1735415896:84 , srv1754792453:84 , srv1762944501:84 , srv1917239806:84 , srv1937063805:84 , srv1944341072:84 , srv1968618711:84 , srv1974734972:84 , srv1978850928:84 , srv1982150796:84 , srv2042434109:84 , srv2050098753:84 , srv2089229867:84 , srv2121487251:84 , srv2137307654:84 , srv2141720134:84 , srv231997951:84 , srv303142508:84 , srv329947076:84 , srv345936083:84 , srv36418608:84 , srv379964049:84 , srv41543449:84 , srv480543460:84 , srv50914240:84 , srv542022774:84 , srv572333656:84 , srv582466234:84 , srv59093425:84 , srv620660338:84 , srv666208248:84 , srv668465979:84 , srv684371613:84 , srv717298218:84 , srv747877195:84 , srv752438543:84 , srv753047405:84 , srv757550758:84 , srv763662732:84 , srv764762110:84 , srv790607647:84 , srv792463969:84 , srv824744380:84 , srv844477396:84 , srv85760024:84 , srv891996884:84 , srv915976715:84 , srv953958555:84 , srv961281890:84 , srv970258704:84 , srv1138363837:85 , srv1200891904:85 , srv1266563183:85 , srv1486113043:85 , srv1543560953:85 , srv1721762127:85 , srv1784674327:85 , srv296255991:85 , srv617861461:85 , srv805850394:85 , srv873384531:85 , srv979758353:85 , srv192635024:684 } 2023-04-16 20:55:40,144 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=377 (was 377), ProcessCount=169 (was 169), AvailableMemoryMB=867 (was 1369) 2023-04-16 20:55:40,155 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=377, ProcessCount=169, AvailableMemoryMB=867 2023-04-16 20:55:40,156 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-16 20:55:40,156 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-04-16 20:55:40,170 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1028780639:84 , srv1031885409:84 , srv1055918603:84 , srv1063157759:84 , srv1277709724:84 , srv1370112609:84 , srv1446854867:84 , srv149024885:84 , srv149926425:84 , srv1528837661:84 , srv1578348700:84 , srv1617374354:84 , srv167140593:84 , srv1692689432:84 , srv1943626825:84 , srv2006298841:84 , srv2037116268:84 , srv2080423424:84 , srv22821530:84 , srv318880784:84 , srv387569003:84 , srv461029653:84 , srv500246342:84 , srv54858579:84 , srv853172893:84 , srv934804456:84 , srv93789197:84 , srv944717825:84 , srv973817464:84 , srv1971108754:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-04-16 20:55:40,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 20:55:40,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv54858579=24, srv167140593=12, srv944717825=28, srv973817464=29, srv387569003=21, srv1028780639=0, srv1578348700=10, srv1971108754=15, srv1277709724=4, srv500246342=23, srv149926425=8, srv1055918603=2, srv1617374354=11, srv318880784=20, srv1943626825=14, srv2006298841=16, srv1370112609=5, srv1528837661=9, srv2080423424=18, srv1692689432=13, srv22821530=19, srv1446854867=6, srv853172893=25, srv1031885409=1, srv93789197=27, srv461029653=22, srv149024885=7, srv934804456=26, srv2037116268=17, srv1063157759=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 20:55:40,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 20:55:40,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 20:55:40,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 20:55:40,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 20:55:40,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 20:55:40,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 20:55:40,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 20:55:40,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 20:55:40,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 20:55:40,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 20:55:40,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 20:55:40,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 20:55:40,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 20:55:40,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 20:55:40,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 20:55:40,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 20:55:40,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 20:55:40,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-16 20:55:40,206 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.044319282848697, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:55:40,206 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912332691877379, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.044319282848697, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-04-16 20:57:40,190 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6159471 different iterations. Found a solution that moves 1867 regions; Going from a computed imbalance of 0.9912332691877379 to a new imbalance of 4.3576711383728133E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6914814814814815, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:57:40,196 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1028780639:90 , srv1031885409:90 , srv1055918603:90 , srv1063157759:90 , srv1277709724:90 , srv1370112609:90 , srv1446854867:90 , srv149024885:90 , srv149926425:90 , srv1528837661:90 , srv1578348700:90 , srv1617374354:90 , srv167140593:90 , srv1692689432:90 , srv1943626825:90 , srv1971108754:90 , srv2006298841:90 , srv2037116268:90 , srv2080423424:90 , srv22821530:90 , srv318880784:90 , srv387569003:90 , srv461029653:90 , srv500246342:90 , srv54858579:90 , srv853172893:90 , srv934804456:90 , srv93789197:90 , srv944717825:90 , srv973817464:90 } 2023-04-16 20:57:40,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-16 20:57:40,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv54858579=24, srv167140593=12, srv944717825=28, srv973817464=29, srv387569003=21, srv1028780639=0, srv1578348700=10, srv1971108754=15, srv1277709724=4, srv500246342=23, srv149926425=8, srv1055918603=2, srv1617374354=11, srv318880784=20, srv1943626825=14, srv2006298841=16, srv1370112609=5, srv1528837661=9, srv2080423424=18, srv1692689432=13, srv22821530=19, srv1446854867=6, srv853172893=25, srv1031885409=1, srv93789197=27, srv461029653=22, srv149024885=7, srv934804456=26, srv2037116268=17, srv1063157759=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-16 20:57:40,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-16 20:57:40,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-16 20:57:40,215 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-16 20:57:40,215 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1028780639:90 , srv1031885409:90 , srv1055918603:90 , srv1063157759:90 , srv1277709724:90 , srv1370112609:90 , srv1446854867:90 , srv149024885:90 , srv149926425:90 , srv1528837661:90 , srv1578348700:90 , srv1617374354:90 , srv167140593:90 , srv1692689432:90 , srv1943626825:90 , srv1971108754:90 , srv2006298841:90 , srv2037116268:90 , srv2080423424:90 , srv22821530:90 , srv318880784:90 , srv387569003:90 , srv461029653:90 , srv500246342:90 , srv54858579:90 , srv853172893:90 , srv934804456:90 , srv93789197:90 , srv944717825:90 , srv973817464:90 } 2023-04-16 20:57:40,229 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=289 (was 377), ProcessCount=169 (was 169), AvailableMemoryMB=960 (was 867) - AvailableMemoryMB LEAK? -