2023-05-30 02:53:22,712 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-30 02:53:22,994 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-30 02:53:23,093 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-30 02:53:23,153 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-30 02:53:23,160 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-30 02:53:23,173 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=170, AvailableMemoryMB=4649 2023-05-30 02:53:23,175 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-30 02:53:23,175 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-30 02:53:23,297 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1021014886:84 , srv1026162216:84 , srv1061120848:84 , srv1097914217:84 , srv109821963:84 , srv1100911610:84 , srv113246409:84 , srv1139298588:84 , srv1178679019:84 , srv1207104908:84 , srv1208607902:84 , srv1223579124:84 , srv122591172:84 , srv1232234719:84 , srv1257615010:84 , srv127663228:84 , srv1288131653:84 , srv1330408011:84 , srv1365283807:84 , srv139612871:84 , srv144484611:84 , srv1449800518:84 , srv1507184184:84 , srv1532196790:84 , srv1533584893:84 , srv1549047155:84 , srv1575450533:84 , srv1578468925:84 , srv1593971354:84 , srv1608770990:84 , srv1668076614:84 , srv1693741867:84 , srv1705195269:84 , srv1719015236:84 , srv1750514258:84 , srv1760352953:84 , srv1774379755:84 , srv1834725831:84 , srv1842177994:84 , srv1849548246:84 , srv1883385950:84 , srv1894635989:84 , srv1901477818:84 , srv1927133798:84 , srv1941079277:84 , srv2000843208:84 , srv2065405260:84 , srv2065740605:84 , srv2097850310:84 , srv2099851790:84 , srv2103800054:84 , srv2107921591:84 , srv2140598964:84 , srv2140655276:84 , srv216493637:84 , srv227601618:84 , srv2662699:84 , srv292712029:84 , srv313392685:84 , srv320841681:84 , srv329256259:84 , srv346713050:84 , srv364640399:84 , srv374106940:84 , srv41768170:84 , srv450348019:84 , srv471221308:84 , srv474123471:84 , srv497099896:84 , srv510047688:84 , srv513815852:84 , srv523680285:84 , srv525868867:84 , srv542456358:84 , srv578260610:84 , srv583439619:84 , srv595448340:84 , srv596369290:84 , srv598944000:84 , srv611647323:84 , srv621615283:84 , srv670097797:84 , srv673697140:84 , srv701712237:84 , srv750068842:84 , srv791579301:84 , srv798693144:84 , srv855867433:84 , srv858387820:84 , srv861781201:84 , srv866352205:84 , srv871743347:84 , srv878459716:84 , srv896734794:84 , srv9039369:84 , srv905325821:84 , srv94769769:84 , srv976941892:84 , srv977948067:84 , srv276567867:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-30 02:53:23,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 02:53:23,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv871743347=92, srv127663228=15, srv364640399=63, srv750068842=85, srv1705195269=32, srv1288131653=16, srv525868867=73, srv41768170=65, srv1507184184=22, srv905325821=96, srv866352205=91, srv1330408011=17, srv670097797=82, srv542456358=74, srv1842177994=38, srv1834725831=37, srv1578468925=27, srv1927133798=43, srv611647323=80, srv523680285=72, srv596369290=78, srv109821963=4, srv1208607902=10, srv976941892=98, srv313392685=59, srv144484611=20, srv94769769=97, srv513815852=71, srv1894635989=41, srv1608770990=29, srv2099851790=49, srv227601618=55, srv2097850310=48, srv450348019=66, srv2065740605=47, srv621615283=81, srv595448340=77, srv1021014886=0, srv471221308=67, srv1549047155=25, srv276567867=57, srv1774379755=36, srv2000843208=45, srv1532196790=23, srv598944000=79, srv1257615010=14, srv139612871=19, srv2140598964=52, srv320841681=60, srv1883385950=40, srv1100911610=5, srv2140655276=53, srv2107921591=51, srv791579301=86, srv1533584893=24, srv2662699=56, srv977948067=99, srv673697140=83, srv113246409=6, srv1449800518=21, srv346713050=62, srv1593971354=28, srv583439619=76, srv798693144=87, srv1061120848=2, srv1178679019=8, srv2065405260=46, srv1901477818=42, srv1575450533=26, srv1760352953=35, srv292712029=58, srv122591172=12, srv1668076614=30, srv861781201=90, srv1693741867=31, srv896734794=94, srv9039369=95, srv474123471=68, srv1026162216=1, srv1941079277=44, srv1207104908=9, srv1750514258=34, srv1097914217=3, srv855867433=88, srv858387820=89, srv578260610=75, srv2103800054=50, srv497099896=69, srv1223579124=11, srv510047688=70, srv1849548246=39, srv878459716=93, srv1365283807=18, srv216493637=54, srv1719015236=33, srv329256259=61, srv374106940=64, srv1232234719=13, srv1139298588=7, srv701712237=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 02:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 02:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 02:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 02:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 02:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 02:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 02:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 02:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 02:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 02:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 02:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 02:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 02:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 02:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 02:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 02:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 02:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 02:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 02:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 02:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 02:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 02:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 02:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 02:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 02:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 02:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 02:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 02:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 02:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 02:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 02:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-30 02:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-30 02:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-30 02:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-30 02:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-30 02:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-30 02:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-30 02:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-30 02:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-30 02:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-30 02:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-30 02:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-30 02:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-30 02:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-30 02:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-30 02:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-30 02:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-30 02:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-30 02:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-30 02:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-30 02:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-30 02:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-30 02:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-30 02:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-30 02:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-30 02:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-30 02:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-30 02:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-30 02:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-30 02:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-30 02:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-30 02:53:23,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-30 02:53:23,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-30 02:53:23,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-30 02:53:23,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-30 02:53:23,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-30 02:53:23,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-30 02:53:23,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-30 02:53:23,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-30 02:53:23,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-30 02:53:23,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-30 02:53:23,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-30 02:53:23,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-30 02:53:23,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-30 02:53:23,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-30 02:53:23,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-30 02:53:23,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-30 02:53:23,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-30 02:53:23,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-30 02:53:23,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-30 02:53:23,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-30 02:53:23,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-30 02:53:23,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-30 02:53:23,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-30 02:53:23,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-30 02:53:23,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-30 02:53:23,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-30 02:53:23,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-30 02:53:23,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-30 02:53:23,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-30 02:53:23,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-30 02:53:23,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-30 02:53:23,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-30 02:53:23,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-30 02:53:23,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-30 02:53:23,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-30 02:53:23,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-30 02:53:23,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-30 02:53:23,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-30 02:53:23,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-30 02:53:23,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-30 02:53:23,541 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:53:23,542 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-30 02:53:23,542 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-30 02:53:23,873 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 466 ms to try 5000 different iterations. Found a solution that moves 4689 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17673314958795142. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0416009582899208, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0568357548582143, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.521, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0416009582899208, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17208333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23683333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:53:23,884 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv144484611:82 , srv701712237:82 , srv122591172:83 , srv2065740605:83 , srv510047688:83 , srv583439619:83 , srv611647323:83 , srv670097797:83 , srv866352205:83 , srv896734794:83 , srv9039369:83 , srv1097914217:84 , srv1178679019:84 , srv1449800518:84 , srv1575450533:84 , srv1593971354:84 , srv1760352953:84 , srv1842177994:84 , srv1894635989:84 , srv2097850310:84 , srv2103800054:84 , srv2662699:84 , srv364640399:84 , srv861781201:84 , srv1061120848:85 , srv109821963:85 , srv127663228:85 , srv139612871:85 , srv1578468925:85 , srv1693741867:85 , srv216493637:85 , srv313392685:85 , srv346713050:85 , srv41768170:85 , srv596369290:85 , srv855867433:85 , srv905325821:85 , srv1021014886:86 , srv113246409:86 , srv1139298588:86 , srv1223579124:86 , srv1288131653:86 , srv1365283807:86 , srv1533584893:86 , srv1668076614:86 , srv1774379755:86 , srv1941079277:86 , srv292712029:86 , srv450348019:86 , srv471221308:86 , srv474123471:86 , srv513815852:86 , srv525868867:86 , srv542456358:86 , srv595448340:86 , srv598944000:86 , srv798693144:86 , srv878459716:86 , srv1207104908:87 , srv1507184184:87 , srv1719015236:87 , srv1750514258:87 , srv1883385950:87 , srv1901477818:87 , srv1927133798:87 , srv2107921591:87 , srv329256259:87 , srv497099896:87 , srv578260610:87 , srv621615283:87 , srv673697140:87 , srv1257615010:88 , srv1330408011:88 , srv1549047155:88 , srv1849548246:88 , srv2065405260:88 , srv227601618:88 , srv374106940:88 , srv791579301:88 , srv858387820:88 , srv1026162216:89 , srv1100911610:89 , srv1208607902:89 , srv1532196790:89 , srv1705195269:89 , srv1834725831:89 , srv2099851790:89 , srv320841681:89 , srv523680285:89 , srv871743347:89 , srv94769769:89 , srv976941892:89 , srv1608770990:90 , srv2000843208:90 , srv2140598964:90 , srv2140655276:90 , srv977948067:90 , srv750068842:91 , srv1232234719:94 , srv276567867:460 } 2023-05-30 02:53:23,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 02:53:23,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv871743347=92, srv127663228=15, srv364640399=63, srv750068842=85, srv1705195269=32, srv1288131653=16, srv525868867=73, srv41768170=65, srv1507184184=22, srv905325821=96, srv866352205=91, srv1330408011=17, srv670097797=82, srv542456358=74, srv1842177994=38, srv1834725831=37, srv1578468925=27, srv1927133798=43, srv611647323=80, srv523680285=72, srv596369290=78, srv109821963=4, srv1208607902=10, srv976941892=98, srv313392685=59, srv144484611=20, srv94769769=97, srv513815852=71, srv1894635989=41, srv1608770990=29, srv2099851790=49, srv227601618=55, srv2097850310=48, srv450348019=66, srv2065740605=47, srv621615283=81, srv595448340=77, srv1021014886=0, srv471221308=67, srv1549047155=25, srv276567867=57, srv1774379755=36, srv2000843208=45, srv1532196790=23, srv598944000=79, srv1257615010=14, srv139612871=19, srv2140598964=52, srv320841681=60, srv1883385950=40, srv1100911610=5, srv2140655276=53, srv2107921591=51, srv791579301=86, srv1533584893=24, srv2662699=56, srv977948067=99, srv673697140=83, srv113246409=6, srv1449800518=21, srv346713050=62, srv1593971354=28, srv583439619=76, srv798693144=87, srv1061120848=2, srv1178679019=8, srv2065405260=46, srv1901477818=42, srv1575450533=26, srv1760352953=35, srv292712029=58, srv122591172=12, srv1668076614=30, srv861781201=90, srv1693741867=31, srv896734794=94, srv9039369=95, srv474123471=68, srv1026162216=1, srv1941079277=44, srv1207104908=9, srv1750514258=34, srv1097914217=3, srv855867433=88, srv858387820=89, srv578260610=75, srv2103800054=50, srv497099896=69, srv1223579124=11, srv510047688=70, srv1849548246=39, srv878459716=93, srv1365283807=18, srv216493637=54, srv1719015236=33, srv329256259=61, srv374106940=64, srv1232234719=13, srv1139298588=7, srv701712237=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 02:53:23,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 02:53:23,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 02:53:23,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 02:53:23,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 02:53:23,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 02:53:23,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 02:53:23,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 02:53:23,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 02:53:23,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 02:53:23,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 02:53:23,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 02:53:23,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 02:53:23,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 02:53:23,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 02:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-30 02:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-30 02:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-30 02:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-30 02:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-30 02:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-30 02:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-30 02:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-30 02:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-30 02:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-30 02:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-30 02:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-30 02:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-30 02:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-30 02:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-30 02:53:24,036 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0416009582899208, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0568357548582143, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0416009582899208, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17208333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23683333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:53:24,037 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-30 02:53:24,037 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17670031650819593, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0416009582899208, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0568357548582143, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0416009582899208, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17208333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23683333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-30 02:53:24,437 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 504 ms to try 5000 different iterations. Found a solution that moves 2637 regions; Going from a computed imbalance of 0.17670031650819593 to a new imbalance of 0.015367873000874928. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023739976455443262); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034530683829044846, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.293, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023739976455443262); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04583333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:53:24,452 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1232234719:77 , srv2140655276:80 , srv1026162216:81 , srv1849548246:81 , srv976941892:81 , srv977948067:81 , srv1208607902:82 , srv1608770990:82 , srv1705195269:82 , srv2000843208:82 , srv2065405260:82 , srv320841681:82 , srv374106940:82 , srv750068842:82 , srv871743347:82 , srv1100911610:83 , srv1257615010:83 , srv1330408011:83 , srv1532196790:83 , srv1750514258:83 , srv1927133798:83 , srv2107921591:83 , srv2140598964:83 , srv1021014886:84 , srv127663228:84 , srv1288131653:84 , srv1533584893:84 , srv292712029:84 , srv471221308:84 , srv523680285:84 , srv798693144:84 , srv858387820:84 , srv94769769:84 , srv1207104908:85 , srv139612871:85 , srv1507184184:85 , srv1693741867:85 , srv1834725831:85 , srv2097850310:85 , srv2099851790:85 , srv227601618:85 , srv41768170:85 , srv525868867:85 , srv621615283:85 , srv673697140:85 , srv905325821:85 , srv1061120848:86 , srv113246409:86 , srv1139298588:86 , srv1668076614:86 , srv1774379755:86 , srv1883385950:86 , srv1941079277:86 , srv313392685:86 , srv346713050:86 , srv450348019:86 , srv513815852:86 , srv542456358:86 , srv595448340:86 , srv791579301:86 , srv1178679019:87 , srv1223579124:87 , srv1365283807:87 , srv1549047155:87 , srv1575450533:87 , srv1593971354:87 , srv474123471:87 , srv497099896:87 , srv578260610:87 , srv109821963:88 , srv122591172:88 , srv1719015236:88 , srv1760352953:88 , srv1901477818:88 , srv2103800054:88 , srv2662699:88 , srv510047688:88 , srv596369290:88 , srv598944000:88 , srv611647323:88 , srv670097797:88 , srv855867433:88 , srv878459716:88 , srv9039369:88 , srv1097914217:89 , srv1449800518:89 , srv1578468925:89 , srv1842177994:89 , srv1894635989:89 , srv2065740605:89 , srv216493637:89 , srv329256259:89 , srv701712237:89 , srv861781201:89 , srv866352205:89 , srv144484611:90 , srv364640399:90 , srv583439619:90 , srv896734794:91 , srv276567867:525 } 2023-05-30 02:53:24,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 02:53:24,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv871743347=92, srv127663228=15, srv364640399=63, srv750068842=85, srv1705195269=32, srv1288131653=16, srv525868867=73, srv41768170=65, srv1507184184=22, srv905325821=96, srv866352205=91, srv1330408011=17, srv670097797=82, srv542456358=74, srv1842177994=38, srv1834725831=37, srv1578468925=27, srv1927133798=43, srv611647323=80, srv523680285=72, srv596369290=78, srv109821963=4, srv1208607902=10, srv976941892=98, srv313392685=59, srv144484611=20, srv94769769=97, srv513815852=71, srv1894635989=41, srv1608770990=29, srv2099851790=49, srv227601618=55, srv2097850310=48, srv450348019=66, srv2065740605=47, srv621615283=81, srv595448340=77, srv1021014886=0, srv471221308=67, srv1549047155=25, srv276567867=57, srv1774379755=36, srv2000843208=45, srv1532196790=23, srv598944000=79, srv1257615010=14, srv139612871=19, srv2140598964=52, srv320841681=60, srv1883385950=40, srv1100911610=5, srv2140655276=53, srv2107921591=51, srv791579301=86, srv1533584893=24, srv2662699=56, srv977948067=99, srv673697140=83, srv113246409=6, srv1449800518=21, srv346713050=62, srv1593971354=28, srv583439619=76, srv798693144=87, srv1061120848=2, srv1178679019=8, srv2065405260=46, srv1901477818=42, srv1575450533=26, srv1760352953=35, srv292712029=58, srv122591172=12, srv1668076614=30, srv861781201=90, srv1693741867=31, srv896734794=94, srv9039369=95, srv474123471=68, srv1026162216=1, srv1941079277=44, srv1207104908=9, srv1750514258=34, srv1097914217=3, srv855867433=88, srv858387820=89, srv578260610=75, srv2103800054=50, srv497099896=69, srv1223579124=11, srv510047688=70, srv1849548246=39, srv878459716=93, srv1365283807=18, srv216493637=54, srv1719015236=33, srv329256259=61, srv374106940=64, srv1232234719=13, srv1139298588=7, srv701712237=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 02:53:24,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 02:53:24,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 02:53:24,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 02:53:24,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 02:53:24,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 02:53:24,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 02:53:24,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 02:53:24,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 02:53:24,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 02:53:24,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 02:53:24,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 02:53:24,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 02:53:24,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 02:53:24,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 02:53:24,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 02:53:24,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 02:53:24,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-30 02:53:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-30 02:53:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-30 02:53:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-30 02:53:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-30 02:53:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-30 02:53:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-30 02:53:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-30 02:53:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-30 02:53:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-30 02:53:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-30 02:53:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-30 02:53:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-30 02:53:24,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-30 02:53:24,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-30 02:53:24,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-30 02:53:24,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-30 02:53:24,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-30 02:53:24,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-30 02:53:24,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-30 02:53:24,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-30 02:53:24,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-30 02:53:24,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-30 02:53:24,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-30 02:53:24,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-30 02:53:24,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-30 02:53:24,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-30 02:53:24,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-30 02:53:24,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-30 02:53:24,645 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023739976455443262); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034530683829044846, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023739976455443262); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04583333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:53:24,645 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-30 02:53:24,645 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015349408332221651, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023739976455443262); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034530683829044846, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023739976455443262); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04583333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-30 02:53:25,025 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 502 ms to try 5000 different iterations. Found a solution that moves 985 regions; Going from a computed imbalance of 0.015349408332221651 to a new imbalance of 0.001065820622902099. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00788367858796349); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008475223015516377); MoveCostFunction : (multiplier=7.0, imbalance=0.10944444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00788367858796349); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:53:25,027 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1549047155:82 , srv1719015236:82 , srv329256259:82 , srv1097914217:83 , srv1578468925:83 , srv1901477818:83 , srv364640399:83 , srv596369290:83 , srv791579301:83 , srv109821963:84 , srv1834725831:84 , srv1842177994:84 , srv1883385950:84 , srv1894635989:84 , srv2099851790:84 , srv2103800054:84 , srv2140598964:84 , srv216493637:84 , srv227601618:84 , srv474123471:84 , srv497099896:84 , srv578260610:84 , srv598944000:84 , srv855867433:84 , srv858387820:84 , srv861781201:84 , srv878459716:84 , srv896734794:84 , srv94769769:84 , srv113246409:85 , srv1139298588:85 , srv1223579124:85 , srv1365283807:85 , srv144484611:85 , srv1449800518:85 , srv1507184184:85 , srv1532196790:85 , srv1608770990:85 , srv1760352953:85 , srv1941079277:85 , srv2000843208:85 , srv2065740605:85 , srv2662699:85 , srv346713050:85 , srv450348019:85 , srv513815852:85 , srv523680285:85 , srv583439619:85 , srv621615283:85 , srv670097797:85 , srv673697140:85 , srv750068842:85 , srv1100911610:86 , srv1178679019:86 , srv1207104908:86 , srv1208607902:86 , srv122591172:86 , srv1232234719:86 , srv1257615010:86 , srv1330408011:86 , srv1575450533:86 , srv1705195269:86 , srv1774379755:86 , srv320841681:86 , srv510047688:86 , srv525868867:86 , srv542456358:86 , srv595448340:86 , srv866352205:86 , srv871743347:86 , srv977948067:86 , srv1021014886:87 , srv1026162216:87 , srv1061120848:87 , srv139612871:87 , srv1533584893:87 , srv1593971354:87 , srv1668076614:87 , srv1693741867:87 , srv1750514258:87 , srv2065405260:87 , srv2107921591:87 , srv2140655276:87 , srv292712029:87 , srv313392685:87 , srv41768170:87 , srv611647323:87 , srv701712237:87 , srv9039369:87 , srv905325821:87 , srv976941892:87 , srv127663228:88 , srv1288131653:88 , srv1849548246:88 , srv1927133798:88 , srv2097850310:88 , srv374106940:88 , srv471221308:88 , srv798693144:88 , srv276567867:543 } 2023-05-30 02:53:25,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 02:53:25,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv871743347=92, srv127663228=15, srv364640399=63, srv750068842=85, srv1705195269=32, srv1288131653=16, srv525868867=73, srv41768170=65, srv1507184184=22, srv905325821=96, srv866352205=91, srv1330408011=17, srv670097797=82, srv542456358=74, srv1842177994=38, srv1834725831=37, srv1578468925=27, srv1927133798=43, srv611647323=80, srv523680285=72, srv596369290=78, srv109821963=4, srv1208607902=10, srv976941892=98, srv313392685=59, srv144484611=20, srv94769769=97, srv513815852=71, srv1894635989=41, srv1608770990=29, srv2099851790=49, srv227601618=55, srv2097850310=48, srv450348019=66, srv2065740605=47, srv621615283=81, srv595448340=77, srv1021014886=0, srv471221308=67, srv1549047155=25, srv276567867=57, srv1774379755=36, srv2000843208=45, srv1532196790=23, srv598944000=79, srv1257615010=14, srv139612871=19, srv2140598964=52, srv320841681=60, srv1883385950=40, srv1100911610=5, srv2140655276=53, srv2107921591=51, srv791579301=86, srv1533584893=24, srv2662699=56, srv977948067=99, srv673697140=83, srv113246409=6, srv1449800518=21, srv346713050=62, srv1593971354=28, srv583439619=76, srv798693144=87, srv1061120848=2, srv1178679019=8, srv2065405260=46, srv1901477818=42, srv1575450533=26, srv1760352953=35, srv292712029=58, srv122591172=12, srv1668076614=30, srv861781201=90, srv1693741867=31, srv896734794=94, srv9039369=95, srv474123471=68, srv1026162216=1, srv1941079277=44, srv1207104908=9, srv1750514258=34, srv1097914217=3, srv855867433=88, srv858387820=89, srv578260610=75, srv2103800054=50, srv497099896=69, srv1223579124=11, srv510047688=70, srv1849548246=39, srv878459716=93, srv1365283807=18, srv216493637=54, srv1719015236=33, srv329256259=61, srv374106940=64, srv1232234719=13, srv1139298588=7, srv701712237=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 02:53:25,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 02:53:25,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 02:53:25,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 02:53:25,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 02:53:25,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 02:53:25,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 02:53:25,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 02:53:25,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 02:53:25,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 02:53:25,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 02:53:25,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 02:53:25,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 02:53:25,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 02:53:25,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 02:53:25,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 02:53:25,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-30 02:53:25,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-30 02:53:25,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-30 02:53:25,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-30 02:53:25,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-30 02:53:25,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-30 02:53:25,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-30 02:53:25,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-30 02:53:25,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-30 02:53:25,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-30 02:53:25,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-30 02:53:25,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-30 02:53:25,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-30 02:53:25,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-30 02:53:25,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-30 02:53:25,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-30 02:53:25,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-30 02:53:25,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-30 02:53:25,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-30 02:53:25,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-30 02:53:25,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-30 02:53:25,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-30 02:53:25,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-30 02:53:25,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-30 02:53:25,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-30 02:53:25,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-30 02:53:25,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-30 02:53:25,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-30 02:53:25,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-30 02:53:25,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-30 02:53:25,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-30 02:53:25,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-30 02:53:25,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-30 02:53:25,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-30 02:53:25,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-30 02:53:25,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-30 02:53:25,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-30 02:53:25,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-30 02:53:25,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-30 02:53:25,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-30 02:53:25,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-30 02:53:25,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-30 02:53:25,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-30 02:53:25,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-30 02:53:25,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-30 02:53:25,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-30 02:53:25,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-30 02:53:25,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-30 02:53:25,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-30 02:53:25,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-30 02:53:25,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-30 02:53:25,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-30 02:53:25,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-30 02:53:25,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-30 02:53:25,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-30 02:53:25,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-30 02:53:25,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-30 02:53:25,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-30 02:53:25,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-30 02:53:25,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-30 02:53:25,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-30 02:53:25,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-30 02:53:25,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-30 02:53:25,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-30 02:53:25,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-30 02:53:25,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-30 02:53:25,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-30 02:53:25,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-30 02:53:25,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-30 02:53:25,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-30 02:53:25,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-30 02:53:25,163 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00788367858796349); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008475223015516377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00788367858796349); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:53:25,164 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-30 02:53:25,164 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0010589235054870524, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00788367858796349); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008475223015516377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00788367858796349); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-30 02:53:25,508 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 441 ms to try 5000 different iterations. Found a solution that moves 301 regions; Going from a computed imbalance of 0.0010589235054870524 to a new imbalance of 4.380585521466154E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.03344444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:53:25,509 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1668076614:83 , srv878459716:83 , srv1061120848:84 , srv1100911610:84 , srv1207104908:84 , srv1223579124:84 , srv1365283807:84 , srv1532196790:84 , srv1593971354:84 , srv1774379755:84 , srv1834725831:84 , srv1901477818:84 , srv1927133798:84 , srv2097850310:84 , srv2099851790:84 , srv2140655276:84 , srv216493637:84 , srv227601618:84 , srv329256259:84 , srv374106940:84 , srv41768170:84 , srv471221308:84 , srv497099896:84 , srv542456358:84 , srv578260610:84 , srv595448340:84 , srv598944000:84 , srv611647323:84 , srv673697140:84 , srv701712237:84 , srv750068842:84 , srv798693144:84 , srv866352205:84 , srv896734794:84 , srv9039369:84 , srv1021014886:85 , srv1026162216:85 , srv109821963:85 , srv113246409:85 , srv1139298588:85 , srv1178679019:85 , srv1208607902:85 , srv122591172:85 , srv1232234719:85 , srv1257615010:85 , srv127663228:85 , srv1288131653:85 , srv1330408011:85 , srv139612871:85 , srv1449800518:85 , srv1507184184:85 , srv1533584893:85 , srv1549047155:85 , srv1575450533:85 , srv1578468925:85 , srv1608770990:85 , srv1693741867:85 , srv1705195269:85 , srv1719015236:85 , srv1750514258:85 , srv1760352953:85 , srv1842177994:85 , srv1849548246:85 , srv1883385950:85 , srv1894635989:85 , srv1941079277:85 , srv2000843208:85 , srv2065405260:85 , srv2065740605:85 , srv2107921591:85 , srv2662699:85 , srv292712029:85 , srv313392685:85 , srv346713050:85 , srv364640399:85 , srv450348019:85 , srv474123471:85 , srv510047688:85 , srv513815852:85 , srv523680285:85 , srv525868867:85 , srv583439619:85 , srv621615283:85 , srv791579301:85 , srv855867433:85 , srv861781201:85 , srv871743347:85 , srv905325821:85 , srv94769769:85 , srv976941892:85 , srv977948067:85 , srv1097914217:86 , srv144484611:86 , srv2103800054:86 , srv2140598964:86 , srv320841681:86 , srv596369290:86 , srv670097797:86 , srv858387820:86 , srv276567867:614 } 2023-05-30 02:53:25,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 02:53:25,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv871743347=92, srv127663228=15, srv364640399=63, srv750068842=85, srv1705195269=32, srv1288131653=16, srv525868867=73, srv41768170=65, srv1507184184=22, srv905325821=96, srv866352205=91, srv1330408011=17, srv670097797=82, srv542456358=74, srv1842177994=38, srv1834725831=37, srv1578468925=27, srv1927133798=43, srv611647323=80, srv523680285=72, srv596369290=78, srv109821963=4, srv1208607902=10, srv976941892=98, srv313392685=59, srv144484611=20, srv94769769=97, srv513815852=71, srv1894635989=41, srv1608770990=29, srv2099851790=49, srv227601618=55, srv2097850310=48, srv450348019=66, srv2065740605=47, srv621615283=81, srv595448340=77, srv1021014886=0, srv471221308=67, srv1549047155=25, srv276567867=57, srv1774379755=36, srv2000843208=45, srv1532196790=23, srv598944000=79, srv1257615010=14, srv139612871=19, srv2140598964=52, srv320841681=60, srv1883385950=40, srv1100911610=5, srv2140655276=53, srv2107921591=51, srv791579301=86, srv1533584893=24, srv2662699=56, srv977948067=99, srv673697140=83, srv113246409=6, srv1449800518=21, srv346713050=62, srv1593971354=28, srv583439619=76, srv798693144=87, srv1061120848=2, srv1178679019=8, srv2065405260=46, srv1901477818=42, srv1575450533=26, srv1760352953=35, srv292712029=58, srv122591172=12, srv1668076614=30, srv861781201=90, srv1693741867=31, srv896734794=94, srv9039369=95, srv474123471=68, srv1026162216=1, srv1941079277=44, srv1207104908=9, srv1750514258=34, srv1097914217=3, srv855867433=88, srv858387820=89, srv578260610=75, srv2103800054=50, srv497099896=69, srv1223579124=11, srv510047688=70, srv1849548246=39, srv878459716=93, srv1365283807=18, srv216493637=54, srv1719015236=33, srv329256259=61, srv374106940=64, srv1232234719=13, srv1139298588=7, srv701712237=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 02:53:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 02:53:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 02:53:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 02:53:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 02:53:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 02:53:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 02:53:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 02:53:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 02:53:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 02:53:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 02:53:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 02:53:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-30 02:53:25,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-30 02:53:25,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-30 02:53:25,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-30 02:53:25,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-30 02:53:25,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-30 02:53:25,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-30 02:53:25,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-30 02:53:25,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-30 02:53:25,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-30 02:53:25,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-30 02:53:25,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-30 02:53:25,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-30 02:53:25,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-30 02:53:25,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-30 02:53:25,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-30 02:53:25,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-30 02:53:25,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-30 02:53:25,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-30 02:53:25,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-30 02:53:25,640 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:53:25,640 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-30 02:53:25,641 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.1698208167017904E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-30 02:53:25,879 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 342 ms to try 5000 different iterations. Found a solution that moves 30 regions; Going from a computed imbalance of 4.1698208167017904E-5 to a new imbalance of 7.899486290149148E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0033333333333333335); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:53:25,879 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1288131653:83 , srv144484611:83 , srv1449800518:83 , srv2140598964:83 , srv313392685:83 , srv320841681:83 , srv523680285:83 , srv583439619:83 , srv1021014886:84 , srv1026162216:84 , srv1061120848:84 , srv1097914217:84 , srv109821963:84 , srv1100911610:84 , srv113246409:84 , srv1139298588:84 , srv1178679019:84 , srv1207104908:84 , srv1208607902:84 , srv1223579124:84 , srv122591172:84 , srv1232234719:84 , srv1257615010:84 , srv127663228:84 , srv1330408011:84 , srv1365283807:84 , srv139612871:84 , srv1507184184:84 , srv1533584893:84 , srv1549047155:84 , srv1575450533:84 , srv1578468925:84 , srv1593971354:84 , srv1608770990:84 , srv1668076614:84 , srv1693741867:84 , srv1705195269:84 , srv1719015236:84 , srv1750514258:84 , srv1760352953:84 , srv1774379755:84 , srv1834725831:84 , srv1842177994:84 , srv1849548246:84 , srv1883385950:84 , srv1894635989:84 , srv1901477818:84 , srv1927133798:84 , srv1941079277:84 , srv2000843208:84 , srv2065405260:84 , srv2065740605:84 , srv2099851790:84 , srv2103800054:84 , srv2107921591:84 , srv216493637:84 , srv2662699:84 , srv292712029:84 , srv329256259:84 , srv364640399:84 , srv374106940:84 , srv450348019:84 , srv471221308:84 , srv474123471:84 , srv497099896:84 , srv510047688:84 , srv513815852:84 , srv525868867:84 , srv542456358:84 , srv578260610:84 , srv595448340:84 , srv596369290:84 , srv598944000:84 , srv611647323:84 , srv621615283:84 , srv670097797:84 , srv701712237:84 , srv750068842:84 , srv791579301:84 , srv798693144:84 , srv855867433:84 , srv858387820:84 , srv861781201:84 , srv866352205:84 , srv871743347:84 , srv896734794:84 , srv9039369:84 , srv905325821:84 , srv94769769:84 , srv976941892:84 , srv977948067:84 , srv1532196790:85 , srv2097850310:85 , srv2140655276:85 , srv227601618:85 , srv346713050:85 , srv41768170:85 , srv673697140:85 , srv878459716:85 , srv276567867:684 } 2023-05-30 02:53:25,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 02:53:25,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv871743347=92, srv127663228=15, srv364640399=63, srv750068842=85, srv1705195269=32, srv1288131653=16, srv525868867=73, srv41768170=65, srv1507184184=22, srv905325821=96, srv866352205=91, srv1330408011=17, srv670097797=82, srv542456358=74, srv1842177994=38, srv1834725831=37, srv1578468925=27, srv1927133798=43, srv611647323=80, srv523680285=72, srv596369290=78, srv109821963=4, srv1208607902=10, srv976941892=98, srv313392685=59, srv144484611=20, srv94769769=97, srv513815852=71, srv1894635989=41, srv1608770990=29, srv2099851790=49, srv227601618=55, srv2097850310=48, srv450348019=66, srv2065740605=47, srv621615283=81, srv595448340=77, srv1021014886=0, srv471221308=67, srv1549047155=25, srv276567867=57, srv1774379755=36, srv2000843208=45, srv1532196790=23, srv598944000=79, srv1257615010=14, srv139612871=19, srv2140598964=52, srv320841681=60, srv1883385950=40, srv1100911610=5, srv2140655276=53, srv2107921591=51, srv791579301=86, srv1533584893=24, srv2662699=56, srv977948067=99, srv673697140=83, srv113246409=6, srv1449800518=21, srv346713050=62, srv1593971354=28, srv583439619=76, srv798693144=87, srv1061120848=2, srv1178679019=8, srv2065405260=46, srv1901477818=42, srv1575450533=26, srv1760352953=35, srv292712029=58, srv122591172=12, srv1668076614=30, srv861781201=90, srv1693741867=31, srv896734794=94, srv9039369=95, srv474123471=68, srv1026162216=1, srv1941079277=44, srv1207104908=9, srv1750514258=34, srv1097914217=3, srv855867433=88, srv858387820=89, srv578260610=75, srv2103800054=50, srv497099896=69, srv1223579124=11, srv510047688=70, srv1849548246=39, srv878459716=93, srv1365283807=18, srv216493637=54, srv1719015236=33, srv329256259=61, srv374106940=64, srv1232234719=13, srv1139298588=7, srv701712237=84} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 02:53:25,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 02:53:25,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 02:53:25,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-30 02:53:25,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-30 02:53:25,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-30 02:53:25,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-30 02:53:25,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-30 02:53:26,017 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:53:26,017 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1288131653:83 , srv144484611:83 , srv1449800518:83 , srv2140598964:83 , srv313392685:83 , srv320841681:83 , srv523680285:83 , srv583439619:83 , srv1021014886:84 , srv1026162216:84 , srv1061120848:84 , srv1097914217:84 , srv109821963:84 , srv1100911610:84 , srv113246409:84 , srv1139298588:84 , srv1178679019:84 , srv1207104908:84 , srv1208607902:84 , srv1223579124:84 , srv122591172:84 , srv1232234719:84 , srv1257615010:84 , srv127663228:84 , srv1330408011:84 , srv1365283807:84 , srv139612871:84 , srv1507184184:84 , srv1533584893:84 , srv1549047155:84 , srv1575450533:84 , srv1578468925:84 , srv1593971354:84 , srv1608770990:84 , srv1668076614:84 , srv1693741867:84 , srv1705195269:84 , srv1719015236:84 , srv1750514258:84 , srv1760352953:84 , srv1774379755:84 , srv1834725831:84 , srv1842177994:84 , srv1849548246:84 , srv1883385950:84 , srv1894635989:84 , srv1901477818:84 , srv1927133798:84 , srv1941079277:84 , srv2000843208:84 , srv2065405260:84 , srv2065740605:84 , srv2099851790:84 , srv2103800054:84 , srv2107921591:84 , srv216493637:84 , srv2662699:84 , srv292712029:84 , srv329256259:84 , srv364640399:84 , srv374106940:84 , srv450348019:84 , srv471221308:84 , srv474123471:84 , srv497099896:84 , srv510047688:84 , srv513815852:84 , srv525868867:84 , srv542456358:84 , srv578260610:84 , srv595448340:84 , srv596369290:84 , srv598944000:84 , srv611647323:84 , srv621615283:84 , srv670097797:84 , srv701712237:84 , srv750068842:84 , srv791579301:84 , srv798693144:84 , srv855867433:84 , srv858387820:84 , srv861781201:84 , srv866352205:84 , srv871743347:84 , srv896734794:84 , srv9039369:84 , srv905325821:84 , srv94769769:84 , srv976941892:84 , srv977948067:84 , srv1532196790:85 , srv2097850310:85 , srv2140655276:85 , srv227601618:85 , srv346713050:85 , srv41768170:85 , srv673697140:85 , srv878459716:85 , srv276567867:684 } 2023-05-30 02:53:26,041 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=258 (was 254) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=4131 (was 4649) 2023-05-30 02:53:26,052 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=258, ProcessCount=170, AvailableMemoryMB=4130 2023-05-30 02:53:26,053 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-30 02:53:26,053 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-30 02:53:26,066 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1143267193:84 , srv125981837:84 , srv1293659074:84 , srv137565886:84 , srv1428553914:84 , srv1466167828:84 , srv1600046924:84 , srv1652189975:84 , srv1885720581:84 , srv1947374844:84 , srv200753402:84 , srv2015292358:84 , srv2050618563:84 , srv2083601472:84 , srv2123654816:84 , srv248354573:84 , srv254074439:84 , srv387387274:84 , srv447079751:84 , srv569292171:84 , srv57202843:84 , srv604751555:84 , srv690148112:84 , srv711204008:84 , srv868174317:84 , srv882595214:84 , srv910527021:84 , srv998492058:84 , srv99908821:84 , srv479599422:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-30 02:53:26,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 02:53:26,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv200753402=10, srv447079751=18, srv604751555=22, srv57202843=21, srv479599422=19, srv1143267193=0, srv1600046924=6, srv99908821=29, srv2123654816=14, srv1947374844=9, srv2050618563=12, srv998492058=28, srv2015292358=11, srv1652189975=7, srv1293659074=2, srv248354573=15, srv254074439=16, srv882595214=26, srv690148112=23, srv2083601472=13, srv125981837=1, srv1428553914=4, srv137565886=3, srv711204008=24, srv1466167828=5, srv1885720581=8, srv387387274=17, srv569292171=20, srv910527021=27, srv868174317=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 02:53:26,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 02:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 02:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 02:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 02:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 02:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 02:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 02:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 02:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 02:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 02:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 02:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 02:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 02:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 02:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 02:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 02:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 02:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 02:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 02:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 02:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 02:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 02:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 02:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 02:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 02:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 02:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 02:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 02:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 02:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 02:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-30 02:53:26,118 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.16990138140401, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:53:26,118 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912728396969293, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.16990138140401, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-30 02:55:26,093 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6140860 different iterations. Found a solution that moves 1859 regions; Going from a computed imbalance of 0.9912728396969293 to a new imbalance of 4.338998739279625E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6885185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:55:26,100 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1143267193:90 , srv125981837:90 , srv1293659074:90 , srv137565886:90 , srv1428553914:90 , srv1466167828:90 , srv1600046924:90 , srv1652189975:90 , srv1885720581:90 , srv1947374844:90 , srv200753402:90 , srv2015292358:90 , srv2050618563:90 , srv2083601472:90 , srv2123654816:90 , srv248354573:90 , srv254074439:90 , srv387387274:90 , srv447079751:90 , srv479599422:90 , srv569292171:90 , srv57202843:90 , srv604751555:90 , srv690148112:90 , srv711204008:90 , srv868174317:90 , srv882595214:90 , srv910527021:90 , srv998492058:90 , srv99908821:90 } 2023-05-30 02:55:26,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 02:55:26,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv200753402=10, srv447079751=18, srv604751555=22, srv57202843=21, srv479599422=19, srv1143267193=0, srv1600046924=6, srv99908821=29, srv2123654816=14, srv1947374844=9, srv2050618563=12, srv998492058=28, srv2015292358=11, srv1652189975=7, srv1293659074=2, srv248354573=15, srv254074439=16, srv882595214=26, srv690148112=23, srv2083601472=13, srv125981837=1, srv1428553914=4, srv137565886=3, srv711204008=24, srv1466167828=5, srv1885720581=8, srv387387274=17, srv569292171=20, srv910527021=27, srv868174317=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 02:55:26,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 02:55:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-30 02:55:26,119 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 02:55:26,119 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1143267193:90 , srv125981837:90 , srv1293659074:90 , srv137565886:90 , srv1428553914:90 , srv1466167828:90 , srv1600046924:90 , srv1652189975:90 , srv1885720581:90 , srv1947374844:90 , srv200753402:90 , srv2015292358:90 , srv2050618563:90 , srv2083601472:90 , srv2123654816:90 , srv248354573:90 , srv254074439:90 , srv387387274:90 , srv447079751:90 , srv479599422:90 , srv569292171:90 , srv57202843:90 , srv604751555:90 , srv690148112:90 , srv711204008:90 , srv868174317:90 , srv882595214:90 , srv910527021:90 , srv998492058:90 , srv99908821:90 } 2023-05-30 02:55:26,132 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=250 (was 258), ProcessCount=168 (was 170), AvailableMemoryMB=3932 (was 4130)