2024-01-25 20:52:58,601 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-01-25 20:52:58,842 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-25 20:52:58,935 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-25 20:52:58,990 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-25 20:52:58,996 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-25 20:52:59,007 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=314, ProcessCount=166, AvailableMemoryMB=6974 2024-01-25 20:52:59,009 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-25 20:52:59,009 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-25 20:52:59,114 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv102543472:84 , srv1040770960:84 , srv108190396:84 , srv1119784375:84 , srv1138826129:84 , srv1140792450:84 , srv116329183:84 , srv1166084548:84 , srv1171173196:84 , srv1311882723:84 , srv132749808:84 , srv1329424193:84 , srv1329843262:84 , srv1335352088:84 , srv1346600515:84 , srv1348937378:84 , srv1350149955:84 , srv1362624614:84 , srv1362976357:84 , srv1380352067:84 , srv1407884708:84 , srv1422383507:84 , srv1425536349:84 , srv1431443980:84 , srv1459845122:84 , srv1483768527:84 , srv1484975722:84 , srv1495306345:84 , srv1495441260:84 , srv1510691957:84 , srv1518578307:84 , srv156177375:84 , srv1588891839:84 , srv1593798186:84 , srv161392004:84 , srv1622635470:84 , srv1672487250:84 , srv1673600033:84 , srv1735877199:84 , srv1744240312:84 , srv1772061939:84 , srv1796982851:84 , srv1807154859:84 , srv181954715:84 , srv1833535950:84 , srv1844761088:84 , srv1869981050:84 , srv1919048698:84 , srv1996869266:84 , srv2003030384:84 , srv201051845:84 , srv2013357980:84 , srv2091481287:84 , srv213455475:84 , srv2143406840:84 , srv220079709:84 , srv25291189:84 , srv261496590:84 , srv311751622:84 , srv323571330:84 , srv331914197:84 , srv349303648:84 , srv349633356:84 , srv351586252:84 , srv356314955:84 , srv362452500:84 , srv363955232:84 , srv368507024:84 , srv378452190:84 , srv384790964:84 , srv417345117:84 , srv421352556:84 , srv436126062:84 , srv454825981:84 , srv488073730:84 , srv494627079:84 , srv499838275:84 , srv535893730:84 , srv566914537:84 , srv601773310:84 , srv603957626:84 , srv620903294:84 , srv656421324:84 , srv684453077:84 , srv68989601:84 , srv705828443:84 , srv706519915:84 , srv709418623:84 , srv721511719:84 , srv737809980:84 , srv75297080:84 , srv80035907:84 , srv85689165:84 , srv895788563:84 , srv900958808:84 , srv909573370:84 , srv909834225:84 , srv939306972:84 , srv965385056:84 , srv1303463240:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-01-25 20:52:59,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 20:52:59,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv349303648=62, srv1510691957=30, srv2143406840=55, srv362452500=66, srv220079709=56, srv1040770960=1, srv1483768527=26, srv1672487250=37, srv709418623=88, srv1138826129=4, srv1422383507=22, srv1484975722=27, srv311751622=59, srv965385056=99, srv454825981=74, srv80035907=92, srv603957626=81, srv900958808=95, srv1407884708=21, srv349633356=63, srv363955232=67, srv1593798186=34, srv620903294=82, srv75297080=91, srv737809980=90, srv2013357980=52, srv351586252=64, srv1495441260=29, srv1311882723=10, srv1844761088=46, srv378452190=69, srv1303463240=9, srv1622635470=36, srv1996869266=49, srv566914537=79, srv1166084548=7, srv499838275=77, srv116329183=6, srv895788563=94, srv2091481287=53, srv939306972=98, srv156177375=32, srv1735877199=39, srv356314955=65, srv1329843262=13, srv1431443980=24, srv494627079=76, srv181954715=44, srv1350149955=17, srv1744240312=40, srv488073730=75, srv1171173196=8, srv1459845122=25, srv331914197=61, srv368507024=68, srv1362624614=18, srv909573370=96, srv1518578307=31, srv2003030384=50, srv909834225=97, srv601773310=80, srv1119784375=3, srv417345117=71, srv1140792450=5, srv102543472=0, srv436126062=73, srv705828443=86, srv1425536349=23, srv684453077=84, srv1588891839=33, srv1673600033=38, srv108190396=2, srv323571330=60, srv1495306345=28, srv535893730=78, srv384790964=70, srv1380352067=20, srv1362976357=19, srv1346600515=15, srv706519915=87, srv132749808=11, srv1919048698=48, srv1807154859=43, srv25291189=57, srv1329424193=12, srv656421324=83, srv161392004=35, srv261496590=58, srv201051845=51, srv421352556=72, srv1833535950=45, srv1869981050=47, srv213455475=54, srv1348937378=16, srv721511719=89, srv85689165=93, srv1796982851=42, srv1335352088=14, srv1772061939=41, srv68989601=85} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 20:52:59,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 20:52:59,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 20:52:59,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 20:52:59,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 20:52:59,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 20:52:59,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 20:52:59,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 20:52:59,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 20:52:59,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 20:52:59,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 20:52:59,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 20:52:59,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 20:52:59,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 20:52:59,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 20:52:59,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 20:52:59,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 20:52:59,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 20:52:59,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 20:52:59,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 20:52:59,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 20:52:59,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 20:52:59,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 20:52:59,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 20:52:59,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 20:52:59,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 20:52:59,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 20:52:59,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 20:52:59,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 20:52:59,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 20:52:59,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 20:52:59,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-25 20:52:59,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-25 20:52:59,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-25 20:52:59,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-25 20:52:59,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-25 20:52:59,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-25 20:52:59,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-25 20:52:59,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-25 20:52:59,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-25 20:52:59,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-25 20:52:59,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-25 20:52:59,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-25 20:52:59,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-25 20:52:59,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-25 20:52:59,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-25 20:52:59,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-25 20:52:59,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-25 20:52:59,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-25 20:52:59,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-25 20:52:59,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-25 20:52:59,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-25 20:52:59,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-25 20:52:59,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-25 20:52:59,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-25 20:52:59,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-25 20:52:59,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-25 20:52:59,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-25 20:52:59,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-25 20:52:59,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-25 20:52:59,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-25 20:52:59,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-25 20:52:59,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-25 20:52:59,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-25 20:52:59,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-25 20:52:59,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-25 20:52:59,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-25 20:52:59,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-25 20:52:59,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-25 20:52:59,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-25 20:52:59,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-25 20:52:59,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-25 20:52:59,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-25 20:52:59,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-25 20:52:59,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-25 20:52:59,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-25 20:52:59,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-25 20:52:59,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-25 20:52:59,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-25 20:52:59,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-25 20:52:59,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-25 20:52:59,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-25 20:52:59,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-25 20:52:59,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-25 20:52:59,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-25 20:52:59,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-25 20:52:59,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-25 20:52:59,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-25 20:52:59,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-25 20:52:59,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-25 20:52:59,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-25 20:52:59,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-25 20:52:59,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-25 20:52:59,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-25 20:52:59,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-25 20:52:59,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-25 20:52:59,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-25 20:52:59,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-25 20:52:59,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-25 20:52:59,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-25 20:52:59,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-25 20:52:59,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-25 20:52:59,429 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 20:52:59,429 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-25 20:52:59,429 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-25 20:52:59,682 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 464 ms to try 5000 different iterations. Found a solution that moves 4695 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17905614910003598. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04071103906599715, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05384226870210707, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5216666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04071103906599715, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1745, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23866666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 20:52:59,691 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1329843262:82 , srv102543472:83 , srv1335352088:83 , srv1362976357:83 , srv1422383507:83 , srv1425536349:83 , srv1495306345:83 , srv1673600033:83 , srv181954715:83 , srv201051845:83 , srv417345117:83 , srv421352556:83 , srv706519915:83 , srv80035907:83 , srv108190396:84 , srv1431443980:84 , srv1622635470:84 , srv1869981050:84 , srv331914197:84 , srv368507024:84 , srv378452190:84 , srv68989601:84 , srv737809980:84 , srv1166084548:85 , srv1329424193:85 , srv1346600515:85 , srv1380352067:85 , srv1483768527:85 , srv1510691957:85 , srv1996869266:85 , srv349303648:85 , srv349633356:85 , srv362452500:85 , srv363955232:85 , srv494627079:85 , srv601773310:85 , srv721511719:85 , srv75297080:85 , srv1495441260:86 , srv156177375:86 , srv1593798186:86 , srv1807154859:86 , srv1833535950:86 , srv2013357980:86 , srv2091481287:86 , srv2143406840:86 , srv323571330:86 , srv620903294:86 , srv895788563:86 , srv909573370:86 , srv939306972:86 , srv965385056:86 , srv1119784375:87 , srv116329183:87 , srv1348937378:87 , srv1484975722:87 , srv161392004:87 , srv1744240312:87 , srv1772061939:87 , srv1919048698:87 , srv25291189:87 , srv311751622:87 , srv351586252:87 , srv384790964:87 , srv436126062:87 , srv535893730:87 , srv656421324:87 , srv909834225:87 , srv1171173196:88 , srv1407884708:88 , srv1672487250:88 , srv1735877199:88 , srv1796982851:88 , srv1844761088:88 , srv220079709:88 , srv261496590:88 , srv488073730:88 , srv499838275:88 , srv603957626:88 , srv684453077:88 , srv705828443:88 , srv85689165:88 , srv900958808:88 , srv1040770960:89 , srv1311882723:89 , srv1362624614:89 , srv1459845122:89 , srv1588891839:89 , srv213455475:89 , srv454825981:89 , srv566914537:89 , srv709418623:89 , srv1138826129:90 , srv1140792450:90 , srv1518578307:90 , srv2003030384:91 , srv132749808:92 , srv356314955:92 , srv1350149955:94 , srv1303463240:452 } 2024-01-25 20:52:59,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 20:52:59,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv349303648=62, srv1510691957=30, srv2143406840=55, srv362452500=66, srv220079709=56, srv1040770960=1, srv1483768527=26, srv1672487250=37, srv709418623=88, srv1138826129=4, srv1422383507=22, srv1484975722=27, srv311751622=59, srv965385056=99, srv454825981=74, srv80035907=92, srv603957626=81, srv900958808=95, srv1407884708=21, srv349633356=63, srv363955232=67, srv1593798186=34, srv620903294=82, srv75297080=91, srv737809980=90, srv2013357980=52, srv351586252=64, srv1495441260=29, srv1311882723=10, srv1844761088=46, srv378452190=69, srv1303463240=9, srv1622635470=36, srv1996869266=49, srv566914537=79, srv1166084548=7, srv499838275=77, srv116329183=6, srv895788563=94, srv2091481287=53, srv939306972=98, srv156177375=32, srv1735877199=39, srv356314955=65, srv1329843262=13, srv1431443980=24, srv494627079=76, srv181954715=44, srv1350149955=17, srv1744240312=40, srv488073730=75, srv1171173196=8, srv1459845122=25, srv331914197=61, srv368507024=68, srv1362624614=18, srv909573370=96, srv1518578307=31, srv2003030384=50, srv909834225=97, srv601773310=80, srv1119784375=3, srv417345117=71, srv1140792450=5, srv102543472=0, srv436126062=73, srv705828443=86, srv1425536349=23, srv684453077=84, srv1588891839=33, srv1673600033=38, srv108190396=2, srv323571330=60, srv1495306345=28, srv535893730=78, srv384790964=70, srv1380352067=20, srv1362976357=19, srv1346600515=15, srv706519915=87, srv132749808=11, srv1919048698=48, srv1807154859=43, srv25291189=57, srv1329424193=12, srv656421324=83, srv161392004=35, srv261496590=58, srv201051845=51, srv421352556=72, srv1833535950=45, srv1869981050=47, srv213455475=54, srv1348937378=16, srv721511719=89, srv85689165=93, srv1796982851=42, srv1335352088=14, srv1772061939=41, srv68989601=85} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 20:52:59,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 20:52:59,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 20:52:59,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 20:52:59,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 20:52:59,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 20:52:59,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 20:52:59,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 20:52:59,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 20:52:59,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 20:52:59,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 20:52:59,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 20:52:59,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 20:52:59,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 20:52:59,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 20:52:59,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 20:52:59,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 20:52:59,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 20:52:59,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 20:52:59,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 20:52:59,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 20:52:59,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 20:52:59,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 20:52:59,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 20:52:59,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 20:52:59,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 20:52:59,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-25 20:52:59,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-25 20:52:59,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-25 20:52:59,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-25 20:52:59,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-25 20:52:59,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-25 20:52:59,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-25 20:52:59,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-25 20:52:59,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-25 20:52:59,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-25 20:52:59,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-25 20:52:59,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-25 20:52:59,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-25 20:52:59,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-25 20:52:59,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-25 20:52:59,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-25 20:52:59,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-25 20:52:59,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-25 20:52:59,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-25 20:52:59,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-25 20:52:59,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-25 20:52:59,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-25 20:52:59,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-25 20:52:59,850 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04071103906599715, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05384226870210707, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04071103906599715, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1745, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23866666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 20:52:59,850 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-25 20:52:59,851 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17902327400738252, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04071103906599715, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05384226870210707, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04071103906599715, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1745, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23866666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-25 20:53:00,026 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 288 ms to try 5000 different iterations. Found a solution that moves 2602 regions; Going from a computed imbalance of 0.17902327400738252 to a new imbalance of 0.014862189906803278. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021615165128022033); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02951915431343641, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2891111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021615165128022033); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 20:53:00,035 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1350149955:78 , srv356314955:80 , srv1588891839:81 , srv1311882723:82 , srv1459845122:82 , srv261496590:82 , srv566914537:82 , srv603957626:82 , srv1040770960:83 , srv1672487250:83 , srv1744240312:83 , srv213455475:83 , srv454825981:83 , srv499838275:83 , srv85689165:83 , srv1140792450:84 , srv1171173196:84 , srv1407884708:84 , srv1518578307:84 , srv156177375:84 , srv1593798186:84 , srv1796982851:84 , srv323571330:84 , srv436126062:84 , srv656421324:84 , srv909573370:84 , srv1119784375:85 , srv1138826129:85 , srv1166084548:85 , srv132749808:85 , srv1348937378:85 , srv1362624614:85 , srv1510691957:85 , srv161392004:85 , srv1807154859:85 , srv1919048698:85 , srv2003030384:85 , srv2091481287:85 , srv220079709:85 , srv25291189:85 , srv349633356:85 , srv384790964:85 , srv488073730:85 , srv535893730:85 , srv684453077:85 , srv705828443:85 , srv709418623:85 , srv721511719:85 , srv895788563:85 , srv900958808:85 , srv909834225:85 , srv108190396:86 , srv1329424193:86 , srv1346600515:86 , srv1735877199:86 , srv1844761088:86 , srv1996869266:86 , srv2143406840:86 , srv311751622:86 , srv331914197:86 , srv368507024:86 , srv378452190:86 , srv601773310:86 , srv965385056:86 , srv116329183:87 , srv1380352067:87 , srv1483768527:87 , srv1484975722:87 , srv1772061939:87 , srv1833535950:87 , srv2013357980:87 , srv351586252:87 , srv363955232:87 , srv494627079:87 , srv68989601:87 , srv939306972:87 , srv102543472:88 , srv1335352088:88 , srv1495441260:88 , srv1673600033:88 , srv181954715:88 , srv201051845:88 , srv362452500:88 , srv417345117:88 , srv620903294:88 , srv737809980:88 , srv75297080:88 , srv1422383507:89 , srv1425536349:89 , srv1495306345:89 , srv1622635470:89 , srv1869981050:89 , srv421352556:89 , srv80035907:89 , srv1329843262:90 , srv1362976357:91 , srv1431443980:91 , srv349303648:91 , srv706519915:91 , srv1303463240:514 } 2024-01-25 20:53:00,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 20:53:00,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv349303648=62, srv1510691957=30, srv2143406840=55, srv362452500=66, srv220079709=56, srv1040770960=1, srv1483768527=26, srv1672487250=37, srv709418623=88, srv1138826129=4, srv1422383507=22, srv1484975722=27, srv311751622=59, srv965385056=99, srv454825981=74, srv80035907=92, srv603957626=81, srv900958808=95, srv1407884708=21, srv349633356=63, srv363955232=67, srv1593798186=34, srv620903294=82, srv75297080=91, srv737809980=90, srv2013357980=52, srv351586252=64, srv1495441260=29, srv1311882723=10, srv1844761088=46, srv378452190=69, srv1303463240=9, srv1622635470=36, srv1996869266=49, srv566914537=79, srv1166084548=7, srv499838275=77, srv116329183=6, srv895788563=94, srv2091481287=53, srv939306972=98, srv156177375=32, srv1735877199=39, srv356314955=65, srv1329843262=13, srv1431443980=24, srv494627079=76, srv181954715=44, srv1350149955=17, srv1744240312=40, srv488073730=75, srv1171173196=8, srv1459845122=25, srv331914197=61, srv368507024=68, srv1362624614=18, srv909573370=96, srv1518578307=31, srv2003030384=50, srv909834225=97, srv601773310=80, srv1119784375=3, srv417345117=71, srv1140792450=5, srv102543472=0, srv436126062=73, srv705828443=86, srv1425536349=23, srv684453077=84, srv1588891839=33, srv1673600033=38, srv108190396=2, srv323571330=60, srv1495306345=28, srv535893730=78, srv384790964=70, srv1380352067=20, srv1362976357=19, srv1346600515=15, srv706519915=87, srv132749808=11, srv1919048698=48, srv1807154859=43, srv25291189=57, srv1329424193=12, srv656421324=83, srv161392004=35, srv261496590=58, srv201051845=51, srv421352556=72, srv1833535950=45, srv1869981050=47, srv213455475=54, srv1348937378=16, srv721511719=89, srv85689165=93, srv1796982851=42, srv1335352088=14, srv1772061939=41, srv68989601=85} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 20:53:00,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 20:53:00,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 20:53:00,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 20:53:00,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 20:53:00,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 20:53:00,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 20:53:00,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 20:53:00,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 20:53:00,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 20:53:00,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 20:53:00,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 20:53:00,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 20:53:00,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 20:53:00,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 20:53:00,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 20:53:00,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 20:53:00,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 20:53:00,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-25 20:53:00,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-25 20:53:00,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-25 20:53:00,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-25 20:53:00,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-25 20:53:00,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-25 20:53:00,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-25 20:53:00,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-25 20:53:00,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-25 20:53:00,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-25 20:53:00,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-25 20:53:00,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-25 20:53:00,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-25 20:53:00,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-25 20:53:00,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-25 20:53:00,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-25 20:53:00,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-25 20:53:00,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-25 20:53:00,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-25 20:53:00,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-25 20:53:00,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-25 20:53:00,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-25 20:53:00,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-25 20:53:00,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-25 20:53:00,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-25 20:53:00,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-25 20:53:00,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-25 20:53:00,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-25 20:53:00,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-25 20:53:00,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-25 20:53:00,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-25 20:53:00,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-25 20:53:00,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-25 20:53:00,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-25 20:53:00,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-25 20:53:00,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-25 20:53:00,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-25 20:53:00,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-25 20:53:00,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-25 20:53:00,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-25 20:53:00,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-25 20:53:00,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-25 20:53:00,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-25 20:53:00,144 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021615165128022033); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02951915431343641, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021615165128022033); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 20:53:00,144 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-25 20:53:00,144 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0148439703133881, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021615165128022033); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02951915431343641, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021615165128022033); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-25 20:53:00,295 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 222 ms to try 5000 different iterations. Found a solution that moves 1000 regions; Going from a computed imbalance of 0.0148439703133881 to a new imbalance of 6.990773496123866E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005654561132722081); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006030226891555272); MoveCostFunction : (multiplier=7.0, imbalance=0.1111111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005654561132722081); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 20:53:00,297 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv132749808:82 , srv1431443980:82 , srv2003030384:82 , srv349303648:82 , srv1138826129:83 , srv1140792450:83 , srv116329183:83 , srv1362624614:83 , srv1484975722:83 , srv1772061939:83 , srv1844761088:83 , srv351586252:83 , srv709418623:83 , srv1362976357:84 , srv1518578307:84 , srv1622635470:84 , srv1735877199:84 , srv1869981050:84 , srv220079709:84 , srv362452500:84 , srv488073730:84 , srv620903294:84 , srv706519915:84 , srv75297080:84 , srv900958808:84 , srv1040770960:85 , srv1171173196:85 , srv1350149955:85 , srv1380352067:85 , srv1407884708:85 , srv1425536349:85 , srv1495306345:85 , srv1495441260:85 , srv1796982851:85 , srv1833535950:85 , srv2013357980:85 , srv2143406840:85 , srv25291189:85 , srv311751622:85 , srv356314955:85 , srv384790964:85 , srv421352556:85 , srv494627079:85 , srv684453077:85 , srv705828443:85 , srv909834225:85 , srv965385056:85 , srv102543472:86 , srv1119784375:86 , srv1311882723:86 , srv1329424193:86 , srv1329843262:86 , srv1335352088:86 , srv1346600515:86 , srv1422383507:86 , srv1459845122:86 , srv1483768527:86 , srv161392004:86 , srv1672487250:86 , srv1673600033:86 , srv1919048698:86 , srv1996869266:86 , srv2091481287:86 , srv213455475:86 , srv363955232:86 , srv436126062:86 , srv454825981:86 , srv566914537:86 , srv601773310:86 , srv68989601:86 , srv737809980:86 , srv80035907:86 , srv895788563:86 , srv939306972:86 , srv108190396:87 , srv1348937378:87 , srv1510691957:87 , srv156177375:87 , srv1588891839:87 , srv1593798186:87 , srv1744240312:87 , srv1807154859:87 , srv181954715:87 , srv201051845:87 , srv331914197:87 , srv349633356:87 , srv378452190:87 , srv417345117:87 , srv499838275:87 , srv535893730:87 , srv656421324:87 , srv909573370:87 , srv1166084548:88 , srv261496590:88 , srv323571330:88 , srv368507024:88 , srv603957626:88 , srv85689165:88 , srv721511719:89 , srv1303463240:542 } 2024-01-25 20:53:00,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 20:53:00,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv349303648=62, srv1510691957=30, srv2143406840=55, srv362452500=66, srv220079709=56, srv1040770960=1, srv1483768527=26, srv1672487250=37, srv709418623=88, srv1138826129=4, srv1422383507=22, srv1484975722=27, srv311751622=59, srv965385056=99, srv454825981=74, srv80035907=92, srv603957626=81, srv900958808=95, srv1407884708=21, srv349633356=63, srv363955232=67, srv1593798186=34, srv620903294=82, srv75297080=91, srv737809980=90, srv2013357980=52, srv351586252=64, srv1495441260=29, srv1311882723=10, srv1844761088=46, srv378452190=69, srv1303463240=9, srv1622635470=36, srv1996869266=49, srv566914537=79, srv1166084548=7, srv499838275=77, srv116329183=6, srv895788563=94, srv2091481287=53, srv939306972=98, srv156177375=32, srv1735877199=39, srv356314955=65, srv1329843262=13, srv1431443980=24, srv494627079=76, srv181954715=44, srv1350149955=17, srv1744240312=40, srv488073730=75, srv1171173196=8, srv1459845122=25, srv331914197=61, srv368507024=68, srv1362624614=18, srv909573370=96, srv1518578307=31, srv2003030384=50, srv909834225=97, srv601773310=80, srv1119784375=3, srv417345117=71, srv1140792450=5, srv102543472=0, srv436126062=73, srv705828443=86, srv1425536349=23, srv684453077=84, srv1588891839=33, srv1673600033=38, srv108190396=2, srv323571330=60, srv1495306345=28, srv535893730=78, srv384790964=70, srv1380352067=20, srv1362976357=19, srv1346600515=15, srv706519915=87, srv132749808=11, srv1919048698=48, srv1807154859=43, srv25291189=57, srv1329424193=12, srv656421324=83, srv161392004=35, srv261496590=58, srv201051845=51, srv421352556=72, srv1833535950=45, srv1869981050=47, srv213455475=54, srv1348937378=16, srv721511719=89, srv85689165=93, srv1796982851=42, srv1335352088=14, srv1772061939=41, srv68989601=85} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 20:53:00,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 20:53:00,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 20:53:00,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 20:53:00,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 20:53:00,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 20:53:00,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 20:53:00,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-25 20:53:00,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-25 20:53:00,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-25 20:53:00,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-25 20:53:00,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-25 20:53:00,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-25 20:53:00,393 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005654561132722081); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006030226891555272); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005654561132722081); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 20:53:00,393 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-25 20:53:00,394 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.92075199952441E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005654561132722081); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006030226891555272); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005654561132722081); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-25 20:53:00,610 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 287 ms to try 5000 different iterations. Found a solution that moves 220 regions; Going from a computed imbalance of 6.92075199952441E-4 to a new imbalance of 1.7846146433742368E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.024444444444444446); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 20:53:00,611 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv132749808:83 , srv1348937378:83 , srv1495441260:83 , srv181954715:83 , srv535893730:83 , srv721511719:83 , srv85689165:83 , srv1040770960:84 , srv1119784375:84 , srv1138826129:84 , srv116329183:84 , srv1166084548:84 , srv1329843262:84 , srv1422383507:84 , srv1483768527:84 , srv1518578307:84 , srv1588891839:84 , srv161392004:84 , srv1744240312:84 , srv1772061939:84 , srv1807154859:84 , srv1833535950:84 , srv1844761088:84 , srv1919048698:84 , srv2003030384:84 , srv201051845:84 , srv2013357980:84 , srv2091481287:84 , srv213455475:84 , srv261496590:84 , srv323571330:84 , srv349303648:84 , srv363955232:84 , srv368507024:84 , srv417345117:84 , srv436126062:84 , srv454825981:84 , srv499838275:84 , srv603957626:84 , srv620903294:84 , srv656421324:84 , srv684453077:84 , srv705828443:84 , srv737809980:84 , srv80035907:84 , srv900958808:84 , srv939306972:84 , srv102543472:85 , srv108190396:85 , srv1140792450:85 , srv1171173196:85 , srv1311882723:85 , srv1329424193:85 , srv1335352088:85 , srv1346600515:85 , srv1350149955:85 , srv1362624614:85 , srv1362976357:85 , srv1380352067:85 , srv1407884708:85 , srv1425536349:85 , srv1431443980:85 , srv1459845122:85 , srv1484975722:85 , srv1495306345:85 , srv1510691957:85 , srv156177375:85 , srv1593798186:85 , srv1622635470:85 , srv1672487250:85 , srv1673600033:85 , srv1735877199:85 , srv1869981050:85 , srv1996869266:85 , srv220079709:85 , srv25291189:85 , srv311751622:85 , srv331914197:85 , srv349633356:85 , srv356314955:85 , srv362452500:85 , srv378452190:85 , srv384790964:85 , srv421352556:85 , srv488073730:85 , srv494627079:85 , srv566914537:85 , srv68989601:85 , srv706519915:85 , srv709418623:85 , srv895788563:85 , srv909573370:85 , srv909834225:85 , srv965385056:85 , srv1796982851:86 , srv2143406840:86 , srv351586252:86 , srv601773310:86 , srv75297080:86 , srv1303463240:634 } 2024-01-25 20:53:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 20:53:00,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv349303648=62, srv1510691957=30, srv2143406840=55, srv362452500=66, srv220079709=56, srv1040770960=1, srv1483768527=26, srv1672487250=37, srv709418623=88, srv1138826129=4, srv1422383507=22, srv1484975722=27, srv311751622=59, srv965385056=99, srv454825981=74, srv80035907=92, srv603957626=81, srv900958808=95, srv1407884708=21, srv349633356=63, srv363955232=67, srv1593798186=34, srv620903294=82, srv75297080=91, srv737809980=90, srv2013357980=52, srv351586252=64, srv1495441260=29, srv1311882723=10, srv1844761088=46, srv378452190=69, srv1303463240=9, srv1622635470=36, srv1996869266=49, srv566914537=79, srv1166084548=7, srv499838275=77, srv116329183=6, srv895788563=94, srv2091481287=53, srv939306972=98, srv156177375=32, srv1735877199=39, srv356314955=65, srv1329843262=13, srv1431443980=24, srv494627079=76, srv181954715=44, srv1350149955=17, srv1744240312=40, srv488073730=75, srv1171173196=8, srv1459845122=25, srv331914197=61, srv368507024=68, srv1362624614=18, srv909573370=96, srv1518578307=31, srv2003030384=50, srv909834225=97, srv601773310=80, srv1119784375=3, srv417345117=71, srv1140792450=5, srv102543472=0, srv436126062=73, srv705828443=86, srv1425536349=23, srv684453077=84, srv1588891839=33, srv1673600033=38, srv108190396=2, srv323571330=60, srv1495306345=28, srv535893730=78, srv384790964=70, srv1380352067=20, srv1362976357=19, srv1346600515=15, srv706519915=87, srv132749808=11, srv1919048698=48, srv1807154859=43, srv25291189=57, srv1329424193=12, srv656421324=83, srv161392004=35, srv261496590=58, srv201051845=51, srv421352556=72, srv1833535950=45, srv1869981050=47, srv213455475=54, srv1348937378=16, srv721511719=89, srv85689165=93, srv1796982851=42, srv1335352088=14, srv1772061939=41, srv68989601=85} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 20:53:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 20:53:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 20:53:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 20:53:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 20:53:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 20:53:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 20:53:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 20:53:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 20:53:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 20:53:00,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 20:53:00,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-25 20:53:00,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-25 20:53:00,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-25 20:53:00,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-25 20:53:00,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-25 20:53:00,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-25 20:53:00,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-25 20:53:00,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-25 20:53:00,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-25 20:53:00,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-25 20:53:00,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-25 20:53:00,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-25 20:53:00,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-25 20:53:00,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-25 20:53:00,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-25 20:53:00,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-25 20:53:00,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-25 20:53:00,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-25 20:53:00,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-25 20:53:00,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-25 20:53:00,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-25 20:53:00,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-25 20:53:00,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-25 20:53:00,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-25 20:53:00,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-25 20:53:00,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-25 20:53:00,708 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.6305673508554334E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 20:53:00,708 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv132749808:83 , srv1348937378:83 , srv1495441260:83 , srv181954715:83 , srv535893730:83 , srv721511719:83 , srv85689165:83 , srv1040770960:84 , srv1119784375:84 , srv1138826129:84 , srv116329183:84 , srv1166084548:84 , srv1329843262:84 , srv1422383507:84 , srv1483768527:84 , srv1518578307:84 , srv1588891839:84 , srv161392004:84 , srv1744240312:84 , srv1772061939:84 , srv1807154859:84 , srv1833535950:84 , srv1844761088:84 , srv1919048698:84 , srv2003030384:84 , srv201051845:84 , srv2013357980:84 , srv2091481287:84 , srv213455475:84 , srv261496590:84 , srv323571330:84 , srv349303648:84 , srv363955232:84 , srv368507024:84 , srv417345117:84 , srv436126062:84 , srv454825981:84 , srv499838275:84 , srv603957626:84 , srv620903294:84 , srv656421324:84 , srv684453077:84 , srv705828443:84 , srv737809980:84 , srv80035907:84 , srv900958808:84 , srv939306972:84 , srv102543472:85 , srv108190396:85 , srv1140792450:85 , srv1171173196:85 , srv1311882723:85 , srv1329424193:85 , srv1335352088:85 , srv1346600515:85 , srv1350149955:85 , srv1362624614:85 , srv1362976357:85 , srv1380352067:85 , srv1407884708:85 , srv1425536349:85 , srv1431443980:85 , srv1459845122:85 , srv1484975722:85 , srv1495306345:85 , srv1510691957:85 , srv156177375:85 , srv1593798186:85 , srv1622635470:85 , srv1672487250:85 , srv1673600033:85 , srv1735877199:85 , srv1869981050:85 , srv1996869266:85 , srv220079709:85 , srv25291189:85 , srv311751622:85 , srv331914197:85 , srv349633356:85 , srv356314955:85 , srv362452500:85 , srv378452190:85 , srv384790964:85 , srv421352556:85 , srv488073730:85 , srv494627079:85 , srv566914537:85 , srv68989601:85 , srv706519915:85 , srv709418623:85 , srv895788563:85 , srv909573370:85 , srv909834225:85 , srv965385056:85 , srv1796982851:86 , srv2143406840:86 , srv351586252:86 , srv601773310:86 , srv75297080:86 , srv1303463240:634 } 2024-01-25 20:53:00,731 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=314 (was 314), ProcessCount=166 (was 166), AvailableMemoryMB=6503 (was 6974) 2024-01-25 20:53:00,737 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=314, ProcessCount=166, AvailableMemoryMB=6503 2024-01-25 20:53:00,738 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-25 20:53:00,738 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-01-25 20:53:00,746 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1086397687:84 , srv1379808224:84 , srv1382538021:84 , srv139222841:84 , srv1555348750:84 , srv1585731785:84 , srv1605715907:84 , srv1621268218:84 , srv1627226152:84 , srv1636252839:84 , srv1646278276:84 , srv1648966594:84 , srv1702235755:84 , srv1825618117:84 , srv1825944658:84 , srv1859067054:84 , srv2098647584:84 , srv2122775737:84 , srv249962095:84 , srv2973868:84 , srv331882241:84 , srv471033922:84 , srv482277102:84 , srv56804087:84 , srv576450943:84 , srv705053869:84 , srv76979269:84 , srv81806637:84 , srv97944597:84 , srv343250052:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-01-25 20:53:00,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 20:53:00,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv97944597=29, srv76979269=27, srv1086397687=0, srv1627226152=8, srv1859067054=15, srv471033922=22, srv1648966594=11, srv2098647584=16, srv1382538021=2, srv1605715907=6, srv1636252839=9, srv81806637=28, srv1621268218=7, srv705053869=26, srv1555348750=4, srv2122775737=17, srv1646278276=10, srv576450943=25, srv1585731785=5, srv139222841=3, srv1825944658=14, srv249962095=18, srv482277102=23, srv1379808224=1, srv1702235755=12, srv343250052=21, srv1825618117=13, srv2973868=19, srv331882241=20, srv56804087=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 20:53:00,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 20:53:00,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 20:53:00,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-25 20:53:00,770 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1281303462342989, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 20:53:00,771 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912596777801422, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1281303462342989, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-01-25 20:55:00,762 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9514135 different iterations. Found a solution that moves 1846 regions; Going from a computed imbalance of 0.9912596777801422 to a new imbalance of 4.308656090753194E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6837037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 20:55:00,768 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1086397687:90 , srv1379808224:90 , srv1382538021:90 , srv139222841:90 , srv1555348750:90 , srv1585731785:90 , srv1605715907:90 , srv1621268218:90 , srv1627226152:90 , srv1636252839:90 , srv1646278276:90 , srv1648966594:90 , srv1702235755:90 , srv1825618117:90 , srv1825944658:90 , srv1859067054:90 , srv2098647584:90 , srv2122775737:90 , srv249962095:90 , srv2973868:90 , srv331882241:90 , srv343250052:90 , srv471033922:90 , srv482277102:90 , srv56804087:90 , srv576450943:90 , srv705053869:90 , srv76979269:90 , srv81806637:90 , srv97944597:90 } 2024-01-25 20:55:00,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-25 20:55:00,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv97944597=29, srv76979269=27, srv1086397687=0, srv1627226152=8, srv1859067054=15, srv471033922=22, srv1648966594=11, srv2098647584=16, srv1382538021=2, srv1605715907=6, srv1636252839=9, srv81806637=28, srv1621268218=7, srv705053869=26, srv1555348750=4, srv2122775737=17, srv1646278276=10, srv576450943=25, srv1585731785=5, srv139222841=3, srv1825944658=14, srv249962095=18, srv482277102=23, srv1379808224=1, srv1702235755=12, srv343250052=21, srv1825618117=13, srv2973868=19, srv331882241=20, srv56804087=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-25 20:55:00,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-25 20:55:00,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-25 20:55:00,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-25 20:55:00,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-25 20:55:00,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-25 20:55:00,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-25 20:55:00,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-25 20:55:00,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-25 20:55:00,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-25 20:55:00,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-25 20:55:00,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-25 20:55:00,784 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-25 20:55:00,784 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1086397687:90 , srv1379808224:90 , srv1382538021:90 , srv139222841:90 , srv1555348750:90 , srv1585731785:90 , srv1605715907:90 , srv1621268218:90 , srv1627226152:90 , srv1636252839:90 , srv1646278276:90 , srv1648966594:90 , srv1702235755:90 , srv1825618117:90 , srv1825944658:90 , srv1859067054:90 , srv2098647584:90 , srv2122775737:90 , srv249962095:90 , srv2973868:90 , srv331882241:90 , srv343250052:90 , srv471033922:90 , srv482277102:90 , srv56804087:90 , srv576450943:90 , srv705053869:90 , srv76979269:90 , srv81806637:90 , srv97944597:90 } 2024-01-25 20:55:00,797 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=195 (was 314), ProcessCount=166 (was 166), AvailableMemoryMB=6611 (was 6503) - AvailableMemoryMB LEAK? -