2024-01-06 19:53:17,119 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-01-06 19:53:17,466 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-06 19:53:17,590 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-06 19:53:17,658 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-06 19:53:17,666 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-06 19:53:17,680 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=270, ProcessCount=167, AvailableMemoryMB=7004 2024-01-06 19:53:17,682 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-06 19:53:17,683 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-06 19:53:17,808 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1060314645:84 , srv1065918629:84 , srv1086726022:84 , srv111054412:84 , srv1114634830:84 , srv1157060206:84 , srv11689132:84 , srv1174570290:84 , srv1198291693:84 , srv120326342:84 , srv120940863:84 , srv1228405354:84 , srv1250835319:84 , srv1256299355:84 , srv1273283048:84 , srv1290080258:84 , srv1302991901:84 , srv1303199152:84 , srv1303870675:84 , srv1303880046:84 , srv1309989674:84 , srv1310960354:84 , srv1312675132:84 , srv132485331:84 , srv1380812766:84 , srv1390585461:84 , srv1418375272:84 , srv1423960184:84 , srv1430566068:84 , srv1458863803:84 , srv1478433804:84 , srv1533678402:84 , srv1566885481:84 , srv1581182978:84 , srv1650464179:84 , srv1656185101:84 , srv1676174926:84 , srv1693100780:84 , srv1694199149:84 , srv1711186197:84 , srv1715009573:84 , srv1726561216:84 , srv1729005310:84 , srv178683060:84 , srv1801871257:84 , srv1823026561:84 , srv1825678380:84 , srv1838290154:84 , srv1842802163:84 , srv185981716:84 , srv1900779797:84 , srv1904410776:84 , srv1933198326:84 , srv1933320637:84 , srv1943121228:84 , srv1960893327:84 , srv1972238770:84 , srv200506806:84 , srv2017453110:84 , srv2024120086:84 , srv2029807115:84 , srv2050989832:84 , srv2051941650:84 , srv2054689538:84 , srv2062491487:84 , srv2069655675:84 , srv2120598865:84 , srv2127256292:84 , srv216738927:84 , srv217818504:84 , srv2180517:84 , srv239807038:84 , srv288707515:84 , srv327454664:84 , srv328356074:84 , srv334487528:84 , srv392030889:84 , srv42734392:84 , srv433578316:84 , srv450096130:84 , srv45511922:84 , srv460418992:84 , srv462380755:84 , srv580551333:84 , srv605163781:84 , srv635381627:84 , srv6386070:84 , srv727988844:84 , srv738342102:84 , srv761232509:84 , srv766883821:84 , srv769089548:84 , srv788184962:84 , srv811296880:84 , srv870065940:84 , srv897178613:84 , srv942190885:84 , srv976162035:84 , srv987247212:84 , srv420510087:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-01-06 19:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-06 19:53:17,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1838290154=47, srv2127256292=67, srv392030889=76, srv942190885=97, srv2069655675=65, srv450096130=80, srv200506806=57, srv1458863803=29, srv1390585461=25, srv1478433804=30, srv2024120086=59, srv761232509=90, srv605163781=85, srv1581182978=33, srv1310960354=21, srv45511922=81, srv1676174926=36, srv1711186197=39, srv1933198326=52, srv216738927=68, srv1303870675=18, srv1418375272=26, srv1114634830=4, srv11689132=6, srv120326342=9, srv1650464179=34, srv2120598865=66, srv580551333=84, srv1065918629=1, srv217818504=69, srv327454664=73, srv185981716=49, srv738342102=89, srv769089548=92, srv334487528=75, srv897178613=96, srv2054689538=63, srv788184962=93, srv1380812766=24, srv2051941650=62, srv1656185101=35, srv1694199149=38, srv2062491487=64, srv1086726022=2, srv1533678402=31, srv1198291693=8, srv1290080258=15, srv132485331=23, srv2180517=70, srv811296880=94, srv1250835319=12, srv1303880046=19, srv178683060=43, srv1825678380=46, srv1157060206=5, srv1174570290=7, srv1972238770=56, srv1273283048=14, srv1566885481=32, srv1960893327=55, srv870065940=95, srv1312675132=22, srv1823026561=45, srv1904410776=51, srv1729005310=42, srv462380755=83, srv1060314645=0, srv420510087=77, srv1423960184=27, srv460418992=82, srv1256299355=13, srv635381627=86, srv766883821=91, srv239807038=71, srv1801871257=44, srv2017453110=58, srv1228405354=11, srv120940863=10, srv1933320637=53, srv433578316=79, srv111054412=3, srv1943121228=54, srv42734392=78, srv727988844=88, srv976162035=98, srv328356074=74, srv6386070=87, srv2050989832=61, srv288707515=72, srv987247212=99, srv1302991901=16, srv2029807115=60, srv1715009573=40, srv1693100780=37, srv1430566068=28, srv1900779797=50, srv1726561216=41, srv1842802163=48, srv1309989674=20, srv1303199152=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-06 19:53:17,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-06 19:53:17,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-06 19:53:17,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-06 19:53:17,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-06 19:53:17,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-06 19:53:17,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-06 19:53:17,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-06 19:53:17,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-06 19:53:17,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-06 19:53:17,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-06 19:53:17,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-06 19:53:17,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-06 19:53:17,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-06 19:53:17,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-06 19:53:17,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-06 19:53:17,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-06 19:53:17,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-06 19:53:17,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-06 19:53:17,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-06 19:53:17,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-06 19:53:17,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-06 19:53:17,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-06 19:53:17,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-06 19:53:17,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-06 19:53:17,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-06 19:53:17,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-06 19:53:17,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-06 19:53:17,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-06 19:53:17,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-06 19:53:17,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-06 19:53:17,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-06 19:53:17,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-06 19:53:17,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-06 19:53:17,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-06 19:53:17,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-06 19:53:17,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-06 19:53:17,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-06 19:53:17,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-06 19:53:17,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-06 19:53:17,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-06 19:53:17,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-06 19:53:17,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-06 19:53:17,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-06 19:53:17,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-06 19:53:17,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-06 19:53:17,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-06 19:53:17,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-06 19:53:17,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-06 19:53:17,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-06 19:53:17,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-06 19:53:17,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-06 19:53:17,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-06 19:53:17,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-06 19:53:17,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-06 19:53:17,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-06 19:53:17,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-06 19:53:17,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-06 19:53:17,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-06 19:53:17,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-06 19:53:17,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-06 19:53:17,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-06 19:53:17,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-06 19:53:17,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-06 19:53:17,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-06 19:53:17,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-06 19:53:17,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-06 19:53:17,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-06 19:53:17,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-06 19:53:17,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-06 19:53:17,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-06 19:53:17,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-06 19:53:17,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-06 19:53:17,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-06 19:53:17,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-06 19:53:17,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-06 19:53:17,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-06 19:53:17,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-06 19:53:17,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-06 19:53:17,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-06 19:53:17,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-06 19:53:17,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-06 19:53:17,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-06 19:53:17,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-06 19:53:17,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-06 19:53:17,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-06 19:53:17,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-06 19:53:17,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-06 19:53:17,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-06 19:53:17,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-06 19:53:17,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-06 19:53:17,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-06 19:53:17,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-06 19:53:17,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-06 19:53:17,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-06 19:53:17,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-06 19:53:17,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-06 19:53:17,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-06 19:53:17,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-06 19:53:17,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-06 19:53:17,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-06 19:53:17,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-06 19:53:18,264 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:53:18,264 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-06 19:53:18,265 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-06 19:53:18,708 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 741 ms to try 5000 different iterations. Found a solution that moves 4668 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18128510117018284. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03860163268103567, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05297861839208349, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5186666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03860163268103567, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17641666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:53:18,718 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1676174926:82 , srv1198291693:83 , srv1290080258:83 , srv1312675132:83 , srv1458863803:83 , srv1478433804:83 , srv1825678380:83 , srv1943121228:83 , srv42734392:83 , srv769089548:83 , srv111054412:84 , srv1273283048:84 , srv1302991901:84 , srv1309989674:84 , srv132485331:84 , srv1533678402:84 , srv1581182978:84 , srv1823026561:84 , srv1838290154:84 , srv1900779797:84 , srv2029807115:84 , srv2051941650:84 , srv2120598865:84 , srv239807038:84 , srv45511922:84 , srv462380755:84 , srv811296880:84 , srv1086726022:85 , srv11689132:85 , srv1650464179:85 , srv1656185101:85 , srv178683060:85 , srv1972238770:85 , srv216738927:85 , srv433578316:85 , srv6386070:85 , srv761232509:85 , srv788184962:85 , srv870065940:85 , srv987247212:85 , srv1065918629:86 , srv1174570290:86 , srv1228405354:86 , srv1380812766:86 , srv1715009573:86 , srv185981716:86 , srv1933198326:86 , srv2180517:86 , srv288707515:86 , srv334487528:86 , srv450096130:86 , srv460418992:86 , srv120940863:87 , srv1256299355:87 , srv1303199152:87 , srv1711186197:87 , srv1933320637:87 , srv1960893327:87 , srv200506806:87 , srv2024120086:87 , srv2069655675:87 , srv580551333:87 , srv635381627:87 , srv738342102:87 , srv897178613:87 , srv1310960354:88 , srv1418375272:88 , srv1423960184:88 , srv1566885481:88 , srv1694199149:88 , srv2062491487:88 , srv2127256292:88 , srv727988844:88 , srv976162035:88 , srv1060314645:89 , srv1114634830:89 , srv1157060206:89 , srv1303880046:89 , srv1390585461:89 , srv1430566068:89 , srv1693100780:89 , srv1801871257:89 , srv2050989832:89 , srv217818504:89 , srv327454664:89 , srv605163781:89 , srv766883821:89 , srv1250835319:90 , srv1842802163:90 , srv1904410776:90 , srv2017453110:90 , srv2054689538:90 , srv942190885:90 , srv1303870675:91 , srv1726561216:91 , srv328356074:91 , srv1729005310:92 , srv392030889:93 , srv120326342:95 , srv420510087:433 } 2024-01-06 19:53:18,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-06 19:53:18,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1838290154=47, srv2127256292=67, srv392030889=76, srv942190885=97, srv2069655675=65, srv450096130=80, srv200506806=57, srv1458863803=29, srv1390585461=25, srv1478433804=30, srv2024120086=59, srv761232509=90, srv605163781=85, srv1581182978=33, srv1310960354=21, srv45511922=81, srv1676174926=36, srv1711186197=39, srv1933198326=52, srv216738927=68, srv1303870675=18, srv1418375272=26, srv1114634830=4, srv11689132=6, srv120326342=9, srv1650464179=34, srv2120598865=66, srv580551333=84, srv1065918629=1, srv217818504=69, srv327454664=73, srv185981716=49, srv738342102=89, srv769089548=92, srv334487528=75, srv897178613=96, srv2054689538=63, srv788184962=93, srv1380812766=24, srv2051941650=62, srv1656185101=35, srv1694199149=38, srv2062491487=64, srv1086726022=2, srv1533678402=31, srv1198291693=8, srv1290080258=15, srv132485331=23, srv2180517=70, srv811296880=94, srv1250835319=12, srv1303880046=19, srv178683060=43, srv1825678380=46, srv1157060206=5, srv1174570290=7, srv1972238770=56, srv1273283048=14, srv1566885481=32, srv1960893327=55, srv870065940=95, srv1312675132=22, srv1823026561=45, srv1904410776=51, srv1729005310=42, srv462380755=83, srv1060314645=0, srv420510087=77, srv1423960184=27, srv460418992=82, srv1256299355=13, srv635381627=86, srv766883821=91, srv239807038=71, srv1801871257=44, srv2017453110=58, srv1228405354=11, srv120940863=10, srv1933320637=53, srv433578316=79, srv111054412=3, srv1943121228=54, srv42734392=78, srv727988844=88, srv976162035=98, srv328356074=74, srv6386070=87, srv2050989832=61, srv288707515=72, srv987247212=99, srv1302991901=16, srv2029807115=60, srv1715009573=40, srv1693100780=37, srv1430566068=28, srv1900779797=50, srv1726561216=41, srv1842802163=48, srv1309989674=20, srv1303199152=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-06 19:53:18,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-06 19:53:18,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-06 19:53:18,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-06 19:53:18,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-06 19:53:18,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-06 19:53:18,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-06 19:53:18,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-06 19:53:18,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-06 19:53:18,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-06 19:53:18,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-06 19:53:18,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-06 19:53:18,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-06 19:53:18,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-06 19:53:18,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-06 19:53:18,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-06 19:53:18,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-06 19:53:18,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-06 19:53:18,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-06 19:53:18,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-06 19:53:18,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-06 19:53:18,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-06 19:53:18,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-06 19:53:18,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-06 19:53:18,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-06 19:53:18,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-06 19:53:18,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-06 19:53:18,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-06 19:53:18,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-06 19:53:18,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-06 19:53:18,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-06 19:53:18,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-06 19:53:18,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-06 19:53:18,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-06 19:53:18,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-06 19:53:18,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-06 19:53:18,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-06 19:53:18,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-06 19:53:18,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-06 19:53:18,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-06 19:53:18,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-06 19:53:18,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-06 19:53:18,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-06 19:53:18,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-06 19:53:18,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-06 19:53:18,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-06 19:53:18,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-06 19:53:18,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-06 19:53:18,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-06 19:53:18,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-06 19:53:18,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-06 19:53:18,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-06 19:53:18,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-06 19:53:18,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-06 19:53:18,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-06 19:53:18,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-06 19:53:18,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-06 19:53:18,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-06 19:53:18,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-06 19:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-06 19:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-06 19:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-06 19:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-06 19:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-06 19:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-06 19:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-06 19:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-06 19:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-06 19:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-06 19:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-06 19:53:18,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-06 19:53:18,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-06 19:53:18,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-06 19:53:18,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-06 19:53:18,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-06 19:53:18,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-06 19:53:18,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-06 19:53:18,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-06 19:53:18,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-06 19:53:18,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-06 19:53:18,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-06 19:53:18,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-06 19:53:18,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-06 19:53:18,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-06 19:53:18,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-06 19:53:18,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-06 19:53:18,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-06 19:53:18,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-06 19:53:18,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-06 19:53:18,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-06 19:53:18,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-06 19:53:18,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-06 19:53:18,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-06 19:53:18,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-06 19:53:18,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-06 19:53:18,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-06 19:53:18,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-06 19:53:18,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-06 19:53:18,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-06 19:53:18,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-06 19:53:18,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-06 19:53:18,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-06 19:53:18,885 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03860163268103567, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05297861839208349, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03860163268103567, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17641666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:53:18,886 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-06 19:53:18,886 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1812524151355702, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03860163268103567, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05297861839208349, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03860163268103567, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17641666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-06 19:53:19,196 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 413 ms to try 5000 different iterations. Found a solution that moves 2690 regions; Going from a computed imbalance of 0.1812524151355702 to a new imbalance of 0.013695182091548623. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02014413705369653); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02836747794065952, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29888888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02014413705369653); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:53:19,209 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv120326342:77 , srv1726561216:80 , srv1904410776:80 , srv1060314645:81 , srv1303870675:81 , srv1303880046:81 , srv1729005310:81 , srv2054689538:81 , srv217818504:81 , srv328356074:81 , srv392030889:81 , srv1157060206:82 , srv1250835319:82 , srv1390585461:82 , srv1842802163:82 , srv2050989832:82 , srv605163781:82 , srv727988844:82 , srv942190885:82 , srv1114634830:83 , srv1310960354:83 , srv1418375272:83 , srv1566885481:83 , srv1801871257:83 , srv2017453110:83 , srv2127256292:83 , srv327454664:83 , srv580551333:83 , srv1174570290:84 , srv1228405354:84 , srv1303199152:84 , srv1430566068:84 , srv1694199149:84 , srv2062491487:84 , srv635381627:84 , srv897178613:84 , srv1065918629:85 , srv120940863:85 , srv1380812766:85 , srv1693100780:85 , srv1715009573:85 , srv200506806:85 , srv2069655675:85 , srv2180517:85 , srv766883821:85 , srv1256299355:86 , srv1423960184:86 , srv1711186197:86 , srv1933198326:86 , srv216738927:86 , srv288707515:86 , srv450096130:86 , srv45511922:86 , srv460418992:86 , srv738342102:86 , srv976162035:86 , srv132485331:87 , srv1823026561:87 , srv1960893327:87 , srv334487528:87 , srv433578316:87 , srv769089548:87 , srv788184962:87 , srv870065940:87 , srv987247212:87 , srv1086726022:88 , srv11689132:88 , srv1273283048:88 , srv1309989674:88 , srv1650464179:88 , srv1656185101:88 , srv1825678380:88 , srv1933320637:88 , srv1972238770:88 , srv2024120086:88 , srv2051941650:88 , srv462380755:88 , srv6386070:88 , srv811296880:88 , srv1198291693:89 , srv1290080258:89 , srv1478433804:89 , srv1533678402:89 , srv178683060:89 , srv185981716:89 , srv1900779797:89 , srv1943121228:89 , srv239807038:89 , srv42734392:89 , srv761232509:89 , srv111054412:90 , srv1302991901:90 , srv1458863803:90 , srv1581182978:90 , srv2029807115:90 , srv2120598865:90 , srv1838290154:91 , srv1312675132:92 , srv1676174926:92 , srv420510087:520 } 2024-01-06 19:53:19,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-06 19:53:19,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1838290154=47, srv2127256292=67, srv392030889=76, srv942190885=97, srv2069655675=65, srv450096130=80, srv200506806=57, srv1458863803=29, srv1390585461=25, srv1478433804=30, srv2024120086=59, srv761232509=90, srv605163781=85, srv1581182978=33, srv1310960354=21, srv45511922=81, srv1676174926=36, srv1711186197=39, srv1933198326=52, srv216738927=68, srv1303870675=18, srv1418375272=26, srv1114634830=4, srv11689132=6, srv120326342=9, srv1650464179=34, srv2120598865=66, srv580551333=84, srv1065918629=1, srv217818504=69, srv327454664=73, srv185981716=49, srv738342102=89, srv769089548=92, srv334487528=75, srv897178613=96, srv2054689538=63, srv788184962=93, srv1380812766=24, srv2051941650=62, srv1656185101=35, srv1694199149=38, srv2062491487=64, srv1086726022=2, srv1533678402=31, srv1198291693=8, srv1290080258=15, srv132485331=23, srv2180517=70, srv811296880=94, srv1250835319=12, srv1303880046=19, srv178683060=43, srv1825678380=46, srv1157060206=5, srv1174570290=7, srv1972238770=56, srv1273283048=14, srv1566885481=32, srv1960893327=55, srv870065940=95, srv1312675132=22, srv1823026561=45, srv1904410776=51, srv1729005310=42, srv462380755=83, srv1060314645=0, srv420510087=77, srv1423960184=27, srv460418992=82, srv1256299355=13, srv635381627=86, srv766883821=91, srv239807038=71, srv1801871257=44, srv2017453110=58, srv1228405354=11, srv120940863=10, srv1933320637=53, srv433578316=79, srv111054412=3, srv1943121228=54, srv42734392=78, srv727988844=88, srv976162035=98, srv328356074=74, srv6386070=87, srv2050989832=61, srv288707515=72, srv987247212=99, srv1302991901=16, srv2029807115=60, srv1715009573=40, srv1693100780=37, srv1430566068=28, srv1900779797=50, srv1726561216=41, srv1842802163=48, srv1309989674=20, srv1303199152=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-06 19:53:19,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-06 19:53:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-06 19:53:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-06 19:53:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-06 19:53:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-06 19:53:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-06 19:53:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-06 19:53:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-06 19:53:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-06 19:53:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-06 19:53:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-06 19:53:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-06 19:53:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-06 19:53:19,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-06 19:53:19,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-06 19:53:19,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-06 19:53:19,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-06 19:53:19,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-06 19:53:19,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-06 19:53:19,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-06 19:53:19,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-06 19:53:19,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-06 19:53:19,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-06 19:53:19,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-06 19:53:19,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-06 19:53:19,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-06 19:53:19,347 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02014413705369653); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02836747794065952, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02014413705369653); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:53:19,348 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-06 19:53:19,348 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013676346308963368, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02014413705369653); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02836747794065952, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02014413705369653); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-06 19:53:19,586 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 336 ms to try 5000 different iterations. Found a solution that moves 983 regions; Going from a computed imbalance of 0.013676346308963368 to a new imbalance of 7.235707743512839E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004319223625811333); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004593461414505888); MoveCostFunction : (multiplier=7.0, imbalance=0.10922222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004319223625811333); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:53:19,589 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1312675132:82 , srv1933320637:82 , srv1423960184:83 , srv1581182978:83 , srv1838290154:83 , srv185981716:83 , srv392030889:83 , srv761232509:83 , srv1302991901:84 , srv1458863803:84 , srv1533678402:84 , srv1650464179:84 , srv1656185101:84 , srv1693100780:84 , srv178683060:84 , srv1900779797:84 , srv1960893327:84 , srv2024120086:84 , srv2029807115:84 , srv2120598865:84 , srv766883821:84 , srv976162035:84 , srv1086726022:85 , srv111054412:85 , srv11689132:85 , srv1198291693:85 , srv120326342:85 , srv120940863:85 , srv1250835319:85 , srv1256299355:85 , srv1430566068:85 , srv1676174926:85 , srv1694199149:85 , srv1711186197:85 , srv1729005310:85 , srv1801871257:85 , srv1943121228:85 , srv1972238770:85 , srv2017453110:85 , srv2069655675:85 , srv239807038:85 , srv288707515:85 , srv328356074:85 , srv334487528:85 , srv42734392:85 , srv460418992:85 , srv6386070:85 , srv738342102:85 , srv788184962:85 , srv811296880:85 , srv870065940:85 , srv1114634830:86 , srv1157060206:86 , srv1273283048:86 , srv1290080258:86 , srv1303199152:86 , srv1303870675:86 , srv1309989674:86 , srv1310960354:86 , srv132485331:86 , srv1380812766:86 , srv1418375272:86 , srv1478433804:86 , srv1726561216:86 , srv1825678380:86 , srv1842802163:86 , srv200506806:86 , srv2051941650:86 , srv2054689538:86 , srv2062491487:86 , srv2127256292:86 , srv216738927:86 , srv2180517:86 , srv327454664:86 , srv433578316:86 , srv450096130:86 , srv462380755:86 , srv605163781:86 , srv635381627:86 , srv942190885:86 , srv1060314645:87 , srv1065918629:87 , srv1390585461:87 , srv1566885481:87 , srv1715009573:87 , srv1823026561:87 , srv1933198326:87 , srv2050989832:87 , srv45511922:87 , srv580551333:87 , srv727988844:87 , srv769089548:87 , srv897178613:87 , srv987247212:87 , srv1174570290:88 , srv1303880046:88 , srv1904410776:88 , srv1228405354:89 , srv217818504:89 , srv420510087:543 } 2024-01-06 19:53:19,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-06 19:53:19,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1838290154=47, srv2127256292=67, srv392030889=76, srv942190885=97, srv2069655675=65, srv450096130=80, srv200506806=57, srv1458863803=29, srv1390585461=25, srv1478433804=30, srv2024120086=59, srv761232509=90, srv605163781=85, srv1581182978=33, srv1310960354=21, srv45511922=81, srv1676174926=36, srv1711186197=39, srv1933198326=52, srv216738927=68, srv1303870675=18, srv1418375272=26, srv1114634830=4, srv11689132=6, srv120326342=9, srv1650464179=34, srv2120598865=66, srv580551333=84, srv1065918629=1, srv217818504=69, srv327454664=73, srv185981716=49, srv738342102=89, srv769089548=92, srv334487528=75, srv897178613=96, srv2054689538=63, srv788184962=93, srv1380812766=24, srv2051941650=62, srv1656185101=35, srv1694199149=38, srv2062491487=64, srv1086726022=2, srv1533678402=31, srv1198291693=8, srv1290080258=15, srv132485331=23, srv2180517=70, srv811296880=94, srv1250835319=12, srv1303880046=19, srv178683060=43, srv1825678380=46, srv1157060206=5, srv1174570290=7, srv1972238770=56, srv1273283048=14, srv1566885481=32, srv1960893327=55, srv870065940=95, srv1312675132=22, srv1823026561=45, srv1904410776=51, srv1729005310=42, srv462380755=83, srv1060314645=0, srv420510087=77, srv1423960184=27, srv460418992=82, srv1256299355=13, srv635381627=86, srv766883821=91, srv239807038=71, srv1801871257=44, srv2017453110=58, srv1228405354=11, srv120940863=10, srv1933320637=53, srv433578316=79, srv111054412=3, srv1943121228=54, srv42734392=78, srv727988844=88, srv976162035=98, srv328356074=74, srv6386070=87, srv2050989832=61, srv288707515=72, srv987247212=99, srv1302991901=16, srv2029807115=60, srv1715009573=40, srv1693100780=37, srv1430566068=28, srv1900779797=50, srv1726561216=41, srv1842802163=48, srv1309989674=20, srv1303199152=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-06 19:53:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-06 19:53:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-06 19:53:19,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-06 19:53:19,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-06 19:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-06 19:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-06 19:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-06 19:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-06 19:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-06 19:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-06 19:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-06 19:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-06 19:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-06 19:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-06 19:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-06 19:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-06 19:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-06 19:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-06 19:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-06 19:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-06 19:53:19,721 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004319223625811333); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004593461414505888); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004319223625811333); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:53:19,722 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-06 19:53:19,722 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.166876612355573E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004319223625811333); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004593461414505888); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004319223625811333); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-06 19:53:20,070 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 451 ms to try 5000 different iterations. Found a solution that moves 193 regions; Going from a computed imbalance of 7.166876612355573E-4 to a new imbalance of 4.061158813504338E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.021444444444444443); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:53:20,072 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv111054412:83 , srv1228405354:83 , srv1676174926:83 , srv1933198326:83 , srv2024120086:83 , srv217818504:83 , srv987247212:83 , srv1065918629:84 , srv1086726022:84 , srv1114634830:84 , srv1174570290:84 , srv120326342:84 , srv1256299355:84 , srv1273283048:84 , srv1290080258:84 , srv1302991901:84 , srv1303870675:84 , srv1303880046:84 , srv1309989674:84 , srv1390585461:84 , srv1423960184:84 , srv1430566068:84 , srv1478433804:84 , srv1566885481:84 , srv1693100780:84 , srv1711186197:84 , srv1715009573:84 , srv1729005310:84 , srv178683060:84 , srv1823026561:84 , srv1838290154:84 , srv1842802163:84 , srv185981716:84 , srv1904410776:84 , srv1943121228:84 , srv1960893327:84 , srv1972238770:84 , srv200506806:84 , srv2017453110:84 , srv2050989832:84 , srv2062491487:84 , srv2120598865:84 , srv239807038:84 , srv288707515:84 , srv327454664:84 , srv334487528:84 , srv433578316:84 , srv450096130:84 , srv635381627:84 , srv6386070:84 , srv738342102:84 , srv766883821:84 , srv788184962:84 , srv897178613:84 , srv942190885:84 , srv976162035:84 , srv1060314645:85 , srv1157060206:85 , srv11689132:85 , srv1198291693:85 , srv120940863:85 , srv1250835319:85 , srv1303199152:85 , srv1310960354:85 , srv1312675132:85 , srv132485331:85 , srv1380812766:85 , srv1418375272:85 , srv1458863803:85 , srv1533678402:85 , srv1650464179:85 , srv1694199149:85 , srv1726561216:85 , srv1801871257:85 , srv1825678380:85 , srv1900779797:85 , srv1933320637:85 , srv2029807115:85 , srv2051941650:85 , srv2054689538:85 , srv2069655675:85 , srv2127256292:85 , srv216738927:85 , srv2180517:85 , srv328356074:85 , srv392030889:85 , srv42734392:85 , srv45511922:85 , srv460418992:85 , srv462380755:85 , srv580551333:85 , srv605163781:85 , srv727988844:85 , srv761232509:85 , srv769089548:85 , srv811296880:85 , srv870065940:85 , srv1581182978:86 , srv1656185101:86 , srv420510087:646 } 2024-01-06 19:53:20,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-06 19:53:20,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1838290154=47, srv2127256292=67, srv392030889=76, srv942190885=97, srv2069655675=65, srv450096130=80, srv200506806=57, srv1458863803=29, srv1390585461=25, srv1478433804=30, srv2024120086=59, srv761232509=90, srv605163781=85, srv1581182978=33, srv1310960354=21, srv45511922=81, srv1676174926=36, srv1711186197=39, srv1933198326=52, srv216738927=68, srv1303870675=18, srv1418375272=26, srv1114634830=4, srv11689132=6, srv120326342=9, srv1650464179=34, srv2120598865=66, srv580551333=84, srv1065918629=1, srv217818504=69, srv327454664=73, srv185981716=49, srv738342102=89, srv769089548=92, srv334487528=75, srv897178613=96, srv2054689538=63, srv788184962=93, srv1380812766=24, srv2051941650=62, srv1656185101=35, srv1694199149=38, srv2062491487=64, srv1086726022=2, srv1533678402=31, srv1198291693=8, srv1290080258=15, srv132485331=23, srv2180517=70, srv811296880=94, srv1250835319=12, srv1303880046=19, srv178683060=43, srv1825678380=46, srv1157060206=5, srv1174570290=7, srv1972238770=56, srv1273283048=14, srv1566885481=32, srv1960893327=55, srv870065940=95, srv1312675132=22, srv1823026561=45, srv1904410776=51, srv1729005310=42, srv462380755=83, srv1060314645=0, srv420510087=77, srv1423960184=27, srv460418992=82, srv1256299355=13, srv635381627=86, srv766883821=91, srv239807038=71, srv1801871257=44, srv2017453110=58, srv1228405354=11, srv120940863=10, srv1933320637=53, srv433578316=79, srv111054412=3, srv1943121228=54, srv42734392=78, srv727988844=88, srv976162035=98, srv328356074=74, srv6386070=87, srv2050989832=61, srv288707515=72, srv987247212=99, srv1302991901=16, srv2029807115=60, srv1715009573=40, srv1693100780=37, srv1430566068=28, srv1900779797=50, srv1726561216=41, srv1842802163=48, srv1309989674=20, srv1303199152=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-06 19:53:20,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-06 19:53:20,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-06 19:53:20,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-06 19:53:20,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-06 19:53:20,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-06 19:53:20,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-06 19:53:20,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-06 19:53:20,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-06 19:53:20,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-06 19:53:20,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-06 19:53:20,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-06 19:53:20,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-06 19:53:20,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-06 19:53:20,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-06 19:53:20,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-06 19:53:20,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-06 19:53:20,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-06 19:53:20,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-06 19:53:20,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-06 19:53:20,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-06 19:53:20,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-06 19:53:20,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-06 19:53:20,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-06 19:53:20,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-06 19:53:20,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-06 19:53:20,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-06 19:53:20,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-06 19:53:20,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-06 19:53:20,199 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:53:20,199 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-06 19:53:20,199 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.9260173250673885E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-06 19:53:20,431 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 328 ms to try 5000 different iterations. Found a solution that moves 20 regions; Going from a computed imbalance of 3.9260173250673885E-5 to a new imbalance of 6.88411204703086E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0022222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:53:20,432 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv11689132:83 , srv1581182978:83 , srv1656185101:83 , srv2029807115:83 , srv2051941650:83 , srv462380755:83 , srv1060314645:84 , srv1065918629:84 , srv1086726022:84 , srv111054412:84 , srv1114634830:84 , srv1157060206:84 , srv1174570290:84 , srv1198291693:84 , srv120940863:84 , srv1228405354:84 , srv1250835319:84 , srv1256299355:84 , srv1273283048:84 , srv1290080258:84 , srv1302991901:84 , srv1303199152:84 , srv1303870675:84 , srv1303880046:84 , srv1309989674:84 , srv1310960354:84 , srv1312675132:84 , srv132485331:84 , srv1380812766:84 , srv1390585461:84 , srv1418375272:84 , srv1430566068:84 , srv1458863803:84 , srv1478433804:84 , srv1533678402:84 , srv1566885481:84 , srv1650464179:84 , srv1676174926:84 , srv1693100780:84 , srv1694199149:84 , srv1711186197:84 , srv1715009573:84 , srv1726561216:84 , srv1729005310:84 , srv178683060:84 , srv1801871257:84 , srv1823026561:84 , srv1825678380:84 , srv1838290154:84 , srv1842802163:84 , srv185981716:84 , srv1900779797:84 , srv1904410776:84 , srv1933198326:84 , srv1933320637:84 , srv1960893327:84 , srv1972238770:84 , srv200506806:84 , srv2017453110:84 , srv2024120086:84 , srv2050989832:84 , srv2054689538:84 , srv2062491487:84 , srv2069655675:84 , srv2120598865:84 , srv2127256292:84 , srv216738927:84 , srv217818504:84 , srv2180517:84 , srv239807038:84 , srv327454664:84 , srv328356074:84 , srv334487528:84 , srv392030889:84 , srv42734392:84 , srv433578316:84 , srv450096130:84 , srv45511922:84 , srv460418992:84 , srv580551333:84 , srv605163781:84 , srv6386070:84 , srv727988844:84 , srv738342102:84 , srv761232509:84 , srv766883821:84 , srv769089548:84 , srv811296880:84 , srv870065940:84 , srv897178613:84 , srv942190885:84 , srv976162035:84 , srv987247212:84 , srv120326342:85 , srv1423960184:85 , srv1943121228:85 , srv288707515:85 , srv635381627:85 , srv788184962:85 , srv420510087:684 } 2024-01-06 19:53:20,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-06 19:53:20,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1838290154=47, srv2127256292=67, srv392030889=76, srv942190885=97, srv2069655675=65, srv450096130=80, srv200506806=57, srv1458863803=29, srv1390585461=25, srv1478433804=30, srv2024120086=59, srv761232509=90, srv605163781=85, srv1581182978=33, srv1310960354=21, srv45511922=81, srv1676174926=36, srv1711186197=39, srv1933198326=52, srv216738927=68, srv1303870675=18, srv1418375272=26, srv1114634830=4, srv11689132=6, srv120326342=9, srv1650464179=34, srv2120598865=66, srv580551333=84, srv1065918629=1, srv217818504=69, srv327454664=73, srv185981716=49, srv738342102=89, srv769089548=92, srv334487528=75, srv897178613=96, srv2054689538=63, srv788184962=93, srv1380812766=24, srv2051941650=62, srv1656185101=35, srv1694199149=38, srv2062491487=64, srv1086726022=2, srv1533678402=31, srv1198291693=8, srv1290080258=15, srv132485331=23, srv2180517=70, srv811296880=94, srv1250835319=12, srv1303880046=19, srv178683060=43, srv1825678380=46, srv1157060206=5, srv1174570290=7, srv1972238770=56, srv1273283048=14, srv1566885481=32, srv1960893327=55, srv870065940=95, srv1312675132=22, srv1823026561=45, srv1904410776=51, srv1729005310=42, srv462380755=83, srv1060314645=0, srv420510087=77, srv1423960184=27, srv460418992=82, srv1256299355=13, srv635381627=86, srv766883821=91, srv239807038=71, srv1801871257=44, srv2017453110=58, srv1228405354=11, srv120940863=10, srv1933320637=53, srv433578316=79, srv111054412=3, srv1943121228=54, srv42734392=78, srv727988844=88, srv976162035=98, srv328356074=74, srv6386070=87, srv2050989832=61, srv288707515=72, srv987247212=99, srv1302991901=16, srv2029807115=60, srv1715009573=40, srv1693100780=37, srv1430566068=28, srv1900779797=50, srv1726561216=41, srv1842802163=48, srv1309989674=20, srv1303199152=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-06 19:53:20,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-06 19:53:20,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-06 19:53:20,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-06 19:53:20,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-06 19:53:20,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-06 19:53:20,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-06 19:53:20,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-06 19:53:20,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-06 19:53:20,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-06 19:53:20,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-06 19:53:20,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-06 19:53:20,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-06 19:53:20,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-06 19:53:20,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-06 19:53:20,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-06 19:53:20,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-06 19:53:20,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-06 19:53:20,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-06 19:53:20,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-06 19:53:20,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-06 19:53:20,566 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:53:20,566 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv11689132:83 , srv1581182978:83 , srv1656185101:83 , srv2029807115:83 , srv2051941650:83 , srv462380755:83 , srv1060314645:84 , srv1065918629:84 , srv1086726022:84 , srv111054412:84 , srv1114634830:84 , srv1157060206:84 , srv1174570290:84 , srv1198291693:84 , srv120940863:84 , srv1228405354:84 , srv1250835319:84 , srv1256299355:84 , srv1273283048:84 , srv1290080258:84 , srv1302991901:84 , srv1303199152:84 , srv1303870675:84 , srv1303880046:84 , srv1309989674:84 , srv1310960354:84 , srv1312675132:84 , srv132485331:84 , srv1380812766:84 , srv1390585461:84 , srv1418375272:84 , srv1430566068:84 , srv1458863803:84 , srv1478433804:84 , srv1533678402:84 , srv1566885481:84 , srv1650464179:84 , srv1676174926:84 , srv1693100780:84 , srv1694199149:84 , srv1711186197:84 , srv1715009573:84 , srv1726561216:84 , srv1729005310:84 , srv178683060:84 , srv1801871257:84 , srv1823026561:84 , srv1825678380:84 , srv1838290154:84 , srv1842802163:84 , srv185981716:84 , srv1900779797:84 , srv1904410776:84 , srv1933198326:84 , srv1933320637:84 , srv1960893327:84 , srv1972238770:84 , srv200506806:84 , srv2017453110:84 , srv2024120086:84 , srv2050989832:84 , srv2054689538:84 , srv2062491487:84 , srv2069655675:84 , srv2120598865:84 , srv2127256292:84 , srv216738927:84 , srv217818504:84 , srv2180517:84 , srv239807038:84 , srv327454664:84 , srv328356074:84 , srv334487528:84 , srv392030889:84 , srv42734392:84 , srv433578316:84 , srv450096130:84 , srv45511922:84 , srv460418992:84 , srv580551333:84 , srv605163781:84 , srv6386070:84 , srv727988844:84 , srv738342102:84 , srv761232509:84 , srv766883821:84 , srv769089548:84 , srv811296880:84 , srv870065940:84 , srv897178613:84 , srv942190885:84 , srv976162035:84 , srv987247212:84 , srv120326342:85 , srv1423960184:85 , srv1943121228:85 , srv288707515:85 , srv635381627:85 , srv788184962:85 , srv420510087:684 } 2024-01-06 19:53:20,592 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=305 (was 270) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=6392 (was 7004) 2024-01-06 19:53:20,600 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=305, ProcessCount=167, AvailableMemoryMB=6392 2024-01-06 19:53:20,600 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-06 19:53:20,601 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-01-06 19:53:20,611 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1031599179:84 , srv1130801543:84 , srv1186051837:84 , srv1348508399:84 , srv1357245144:84 , srv1387406453:84 , srv1438297343:84 , srv1446550110:84 , srv1526512569:84 , srv1566014022:84 , srv1620976623:84 , srv1784213919:84 , srv1899759365:84 , srv190343428:84 , srv285566920:84 , srv290182724:84 , srv320056388:84 , srv367643281:84 , srv408453939:84 , srv462680059:84 , srv559543984:84 , srv607018219:84 , srv630095876:84 , srv641614223:84 , srv713567207:84 , srv844637867:84 , srv88936029:84 , srv89126750:84 , srv969953474:84 , srv1191095156:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-01-06 19:53:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-06 19:53:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv367643281=18, srv844637867=26, srv1566014022=10, srv1446550110=8, srv408453939=19, srv89126750=28, srv1348508399=4, srv462680059=20, srv1620976623=11, srv1130801543=1, srv290182724=16, srv88936029=27, srv1191095156=3, srv1899759365=13, srv190343428=14, srv607018219=22, srv630095876=23, srv1526512569=9, srv1784213919=12, srv1186051837=2, srv713567207=25, srv320056388=17, srv559543984=21, srv1031599179=0, srv1387406453=6, srv1357245144=5, srv285566920=15, srv969953474=29, srv641614223=24, srv1438297343=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-06 19:53:20,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-06 19:53:20,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-06 19:53:20,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-06 19:53:20,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-06 19:53:20,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-06 19:53:20,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-06 19:53:20,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-06 19:53:20,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-06 19:53:20,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-06 19:53:20,640 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1701778024135159, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:53:20,640 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912729267962868, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1701778024135159, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-01-06 19:55:20,628 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5681071 different iterations. Found a solution that moves 1854 regions; Going from a computed imbalance of 0.9912729267962868 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:55:20,634 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1031599179:90 , srv1130801543:90 , srv1186051837:90 , srv1191095156:90 , srv1348508399:90 , srv1357245144:90 , srv1387406453:90 , srv1438297343:90 , srv1446550110:90 , srv1526512569:90 , srv1566014022:90 , srv1620976623:90 , srv1784213919:90 , srv1899759365:90 , srv190343428:90 , srv285566920:90 , srv290182724:90 , srv320056388:90 , srv367643281:90 , srv408453939:90 , srv462680059:90 , srv559543984:90 , srv607018219:90 , srv630095876:90 , srv641614223:90 , srv713567207:90 , srv844637867:90 , srv88936029:90 , srv89126750:90 , srv969953474:90 } 2024-01-06 19:55:20,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-06 19:55:20,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv367643281=18, srv844637867=26, srv1566014022=10, srv1446550110=8, srv408453939=19, srv89126750=28, srv1348508399=4, srv462680059=20, srv1620976623=11, srv1130801543=1, srv290182724=16, srv88936029=27, srv1191095156=3, srv1899759365=13, srv190343428=14, srv607018219=22, srv630095876=23, srv1526512569=9, srv1784213919=12, srv1186051837=2, srv713567207=25, srv320056388=17, srv559543984=21, srv1031599179=0, srv1387406453=6, srv1357245144=5, srv285566920=15, srv969953474=29, srv641614223=24, srv1438297343=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-06 19:55:20,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-06 19:55:20,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-06 19:55:20,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-06 19:55:20,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-06 19:55:20,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-06 19:55:20,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-06 19:55:20,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-06 19:55:20,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-06 19:55:20,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-06 19:55:20,656 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-06 19:55:20,656 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1031599179:90 , srv1130801543:90 , srv1186051837:90 , srv1191095156:90 , srv1348508399:90 , srv1357245144:90 , srv1387406453:90 , srv1438297343:90 , srv1446550110:90 , srv1526512569:90 , srv1566014022:90 , srv1620976623:90 , srv1784213919:90 , srv1899759365:90 , srv190343428:90 , srv285566920:90 , srv290182724:90 , srv320056388:90 , srv367643281:90 , srv408453939:90 , srv462680059:90 , srv559543984:90 , srv607018219:90 , srv630095876:90 , srv641614223:90 , srv713567207:90 , srv844637867:90 , srv88936029:90 , srv89126750:90 , srv969953474:90 } 2024-01-06 19:55:20,669 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=290 (was 305), ProcessCount=167 (was 167), AvailableMemoryMB=5693 (was 6392)