2023-12-16 10:54:18,472 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-16 10:54:18,802 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-16 10:54:18,926 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-16 10:54:18,991 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-16 10:54:18,998 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-16 10:54:19,011 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=321, ProcessCount=168, AvailableMemoryMB=7360 2023-12-16 10:54:19,014 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-16 10:54:19,014 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-16 10:54:19,155 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv103092926:84 , srv1063391143:84 , srv1079418042:84 , srv1098263132:84 , srv1132476308:84 , srv1153931253:84 , srv1211362611:84 , srv1222492841:84 , srv1239750324:84 , srv1239760934:84 , srv1247109217:84 , srv1272116287:84 , srv127953046:84 , srv1295993964:84 , srv1304286216:84 , srv1309644784:84 , srv133202840:84 , srv1353811044:84 , srv1418221971:84 , srv1422175831:84 , srv1433773553:84 , srv1439246241:84 , srv1469859447:84 , srv1483826960:84 , srv1503311047:84 , srv1527648277:84 , srv1566878416:84 , srv1575895823:84 , srv1592801702:84 , srv1601968701:84 , srv162469518:84 , srv1648103647:84 , srv1655307662:84 , srv1728320641:84 , srv1752767683:84 , srv181483074:84 , srv183408682:84 , srv1847714149:84 , srv1855758158:84 , srv1884251959:84 , srv1887536122:84 , srv1894862012:84 , srv1903485773:84 , srv1917939982:84 , srv192352895:84 , srv1951630017:84 , srv1954331564:84 , srv2013722430:84 , srv2081021998:84 , srv2082147938:84 , srv2087923926:84 , srv2094984088:84 , srv2120927230:84 , srv2134612531:84 , srv2134760103:84 , srv241069586:84 , srv256926400:84 , srv257392250:84 , srv258877909:84 , srv293936383:84 , srv334739251:84 , srv34760423:84 , srv366913283:84 , srv370716956:84 , srv388051609:84 , srv415533951:84 , srv427552121:84 , srv428871281:84 , srv434655485:84 , srv44668621:84 , srv459211219:84 , srv471291045:84 , srv508523515:84 , srv537605653:84 , srv55920725:84 , srv56402784:84 , srv587729608:84 , srv588690309:84 , srv597655144:84 , srv611178502:84 , srv611469071:84 , srv6347447:84 , srv634903216:84 , srv645100469:84 , srv652003344:84 , srv686888876:84 , srv68995494:84 , srv704216461:84 , srv716371335:84 , srv736301695:84 , srv7435289:84 , srv75802784:84 , srv775431397:84 , srv798903960:84 , srv842631842:84 , srv851727095:84 , srv856944210:84 , srv944023371:84 , srv952957057:84 , srv1230002842:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-16 10:54:19,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 10:54:19,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv686888876=86, srv1439246241=22, srv181483074=36, srv2082147938=50, srv587729608=77, srv588690309=78, srv1422175831=20, srv1903485773=43, srv127953046=13, srv1304286216=15, srv192352895=45, srv68995494=87, srv1239760934=10, srv1433773553=21, srv1752767683=35, srv851727095=96, srv611469071=81, srv75802784=92, srv944023371=98, srv1153931253=5, srv7435289=91, srv1295993964=14, srv388051609=65, srv1353811044=18, srv1884251959=40, srv1483826960=24, srv1575895823=28, srv6347447=82, srv1951630017=46, srv1230002842=8, srv736301695=90, srv1272116287=12, srv44668621=70, srv162469518=31, srv427552121=67, srv257392250=58, srv293936383=60, srv2134612531=54, srv1917939982=44, srv2120927230=53, srv1592801702=29, srv2081021998=49, srv428871281=68, srv1222492841=7, srv1655307662=33, srv1079418042=2, srv1728320641=34, srv241069586=56, srv856944210=97, srv1247109217=11, srv2134760103=55, srv1239750324=9, srv1418221971=19, srv842631842=95, srv471291045=72, srv597655144=79, srv1847714149=38, srv798903960=94, srv334739251=61, srv1887536122=41, srv2094984088=52, srv704216461=88, srv2087923926=51, srv634903216=83, srv1469859447=23, srv133202840=17, srv1098263132=3, srv1855758158=39, srv2013722430=48, srv1211362611=6, srv366913283=63, srv1309644784=16, srv34760423=62, srv1063391143=1, srv508523515=73, srv1132476308=4, srv952957057=99, srv1527648277=26, srv1503311047=25, srv103092926=0, srv258877909=59, srv370716956=64, srv415533951=66, srv775431397=93, srv537605653=74, srv55920725=75, srv1566878416=27, srv645100469=84, srv459211219=71, srv716371335=89, srv652003344=85, srv1648103647=32, srv434655485=69, srv56402784=76, srv1894862012=42, srv256926400=57, srv1601968701=30, srv183408682=37, srv1954331564=47, srv611178502=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 10:54:19,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 10:54:19,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 10:54:19,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 10:54:19,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 10:54:19,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 10:54:19,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 10:54:19,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 10:54:19,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 10:54:19,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 10:54:19,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 10:54:19,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 10:54:19,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 10:54:19,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 10:54:19,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 10:54:19,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 10:54:19,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 10:54:19,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 10:54:19,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 10:54:19,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 10:54:19,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 10:54:19,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 10:54:19,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 10:54:19,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 10:54:19,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 10:54:19,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 10:54:19,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 10:54:19,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 10:54:19,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 10:54:19,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 10:54:19,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 10:54:19,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-16 10:54:19,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-16 10:54:19,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-16 10:54:19,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-16 10:54:19,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-16 10:54:19,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-16 10:54:19,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-16 10:54:19,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-16 10:54:19,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-16 10:54:19,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-16 10:54:19,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-16 10:54:19,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-16 10:54:19,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-16 10:54:19,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-16 10:54:19,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-16 10:54:19,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-16 10:54:19,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-16 10:54:19,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-16 10:54:19,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-16 10:54:19,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-16 10:54:19,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-16 10:54:19,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-16 10:54:19,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-16 10:54:19,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-16 10:54:19,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-16 10:54:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-16 10:54:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-16 10:54:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-16 10:54:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-16 10:54:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-16 10:54:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-16 10:54:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-16 10:54:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-16 10:54:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-16 10:54:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-16 10:54:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-16 10:54:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-16 10:54:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-16 10:54:19,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-16 10:54:19,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-16 10:54:19,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-16 10:54:19,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-16 10:54:19,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-16 10:54:19,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-16 10:54:19,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-16 10:54:19,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-16 10:54:19,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-16 10:54:19,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-16 10:54:19,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-16 10:54:19,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-16 10:54:19,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-16 10:54:19,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-16 10:54:19,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-16 10:54:19,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-16 10:54:19,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-16 10:54:19,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-16 10:54:19,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-16 10:54:19,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-16 10:54:19,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-16 10:54:19,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-16 10:54:19,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-16 10:54:19,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-16 10:54:19,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-16 10:54:19,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-16 10:54:19,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-16 10:54:19,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-16 10:54:19,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-16 10:54:19,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-16 10:54:19,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-16 10:54:19,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-16 10:54:19,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-16 10:54:19,619 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:54:19,619 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-16 10:54:19,620 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-16 10:54:20,442 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1133 ms to try 5000 different iterations. Found a solution that moves 4664 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1786832679661267. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036911909845130125, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051785190434839715, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5182222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.036911909845130125, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17391666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24066666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:54:20,458 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1655307662:81 , srv1304286216:82 , srv1592801702:82 , srv388051609:82 , srv1153931253:83 , srv415533951:83 , srv459211219:83 , srv856944210:83 , srv944023371:83 , srv1309644784:84 , srv1847714149:84 , srv1951630017:84 , srv2094984088:84 , srv34760423:84 , srv611469071:84 , srv634903216:84 , srv1063391143:85 , srv1239750324:85 , srv133202840:85 , srv1353811044:85 , srv1483826960:85 , srv1566878416:85 , srv1575895823:85 , srv1601968701:85 , srv181483074:85 , srv1887536122:85 , srv2087923926:85 , srv2134612531:85 , srv56402784:85 , srv597655144:85 , srv68995494:85 , srv775431397:85 , srv1098263132:86 , srv127953046:86 , srv162469518:86 , srv1894862012:86 , srv1903485773:86 , srv1954331564:86 , srv2082147938:86 , srv2134760103:86 , srv241069586:86 , srv258877909:86 , srv293936383:86 , srv434655485:86 , srv508523515:86 , srv587729608:86 , srv652003344:86 , srv716371335:86 , srv7435289:86 , srv103092926:87 , srv1211362611:87 , srv1222492841:87 , srv1272116287:87 , srv1418221971:87 , srv1527648277:87 , srv1752767683:87 , srv183408682:87 , srv1855758158:87 , srv192352895:87 , srv2120927230:87 , srv366913283:87 , srv645100469:87 , srv704216461:87 , srv736301695:87 , srv75802784:87 , srv842631842:87 , srv952957057:87 , srv1247109217:88 , srv1469859447:88 , srv1648103647:88 , srv1728320641:88 , srv257392250:88 , srv370716956:88 , srv427552121:88 , srv611178502:88 , srv6347447:88 , srv798903960:88 , srv1239760934:89 , srv1422175831:89 , srv1503311047:89 , srv1917939982:89 , srv2013722430:89 , srv537605653:89 , srv55920725:89 , srv588690309:89 , srv1295993964:90 , srv334739251:90 , srv44668621:90 , srv471291045:90 , srv686888876:90 , srv851727095:90 , srv1079418042:91 , srv1433773553:91 , srv1884251959:91 , srv2081021998:91 , srv256926400:91 , srv1132476308:92 , srv1439246241:92 , srv428871281:93 , srv1230002842:418 } 2023-12-16 10:54:20,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 10:54:20,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv686888876=86, srv1439246241=22, srv181483074=36, srv2082147938=50, srv587729608=77, srv588690309=78, srv1422175831=20, srv1903485773=43, srv127953046=13, srv1304286216=15, srv192352895=45, srv68995494=87, srv1239760934=10, srv1433773553=21, srv1752767683=35, srv851727095=96, srv611469071=81, srv75802784=92, srv944023371=98, srv1153931253=5, srv7435289=91, srv1295993964=14, srv388051609=65, srv1353811044=18, srv1884251959=40, srv1483826960=24, srv1575895823=28, srv6347447=82, srv1951630017=46, srv1230002842=8, srv736301695=90, srv1272116287=12, srv44668621=70, srv162469518=31, srv427552121=67, srv257392250=58, srv293936383=60, srv2134612531=54, srv1917939982=44, srv2120927230=53, srv1592801702=29, srv2081021998=49, srv428871281=68, srv1222492841=7, srv1655307662=33, srv1079418042=2, srv1728320641=34, srv241069586=56, srv856944210=97, srv1247109217=11, srv2134760103=55, srv1239750324=9, srv1418221971=19, srv842631842=95, srv471291045=72, srv597655144=79, srv1847714149=38, srv798903960=94, srv334739251=61, srv1887536122=41, srv2094984088=52, srv704216461=88, srv2087923926=51, srv634903216=83, srv1469859447=23, srv133202840=17, srv1098263132=3, srv1855758158=39, srv2013722430=48, srv1211362611=6, srv366913283=63, srv1309644784=16, srv34760423=62, srv1063391143=1, srv508523515=73, srv1132476308=4, srv952957057=99, srv1527648277=26, srv1503311047=25, srv103092926=0, srv258877909=59, srv370716956=64, srv415533951=66, srv775431397=93, srv537605653=74, srv55920725=75, srv1566878416=27, srv645100469=84, srv459211219=71, srv716371335=89, srv652003344=85, srv1648103647=32, srv434655485=69, srv56402784=76, srv1894862012=42, srv256926400=57, srv1601968701=30, srv183408682=37, srv1954331564=47, srv611178502=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 10:54:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 10:54:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 10:54:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 10:54:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 10:54:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 10:54:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 10:54:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 10:54:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 10:54:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 10:54:20,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 10:54:20,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-16 10:54:20,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-16 10:54:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-16 10:54:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-16 10:54:20,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-16 10:54:20,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-16 10:54:20,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-16 10:54:20,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-16 10:54:20,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-16 10:54:20,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-16 10:54:20,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-16 10:54:20,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-16 10:54:20,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-16 10:54:20,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-16 10:54:20,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-16 10:54:20,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-16 10:54:20,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-16 10:54:20,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-16 10:54:20,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-16 10:54:20,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-16 10:54:20,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-16 10:54:20,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-16 10:54:20,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-16 10:54:20,651 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036911909845130125, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051785190434839715, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.036911909845130125, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17391666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24066666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:54:20,651 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-16 10:54:20,651 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17865060994011273, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036911909845130125, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051785190434839715, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.036911909845130125, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17391666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24066666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-16 10:54:20,937 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 425 ms to try 5000 different iterations. Found a solution that moves 2677 regions; Going from a computed imbalance of 0.17865060994011273 to a new imbalance of 0.014835004712266835. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021376732884530333); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033618310440256566, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29744444444444446, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021376732884530333); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042583333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:54:20,950 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv428871281:77 , srv1132476308:79 , srv334739251:80 , srv851727095:80 , srv1079418042:81 , srv1439246241:81 , srv1503311047:81 , srv1884251959:81 , srv44668621:81 , srv1239760934:82 , srv1917939982:82 , srv256926400:82 , srv1222492841:83 , srv1295993964:83 , srv1422175831:83 , srv2081021998:83 , srv2120927230:83 , srv370716956:83 , srv471291045:83 , srv508523515:83 , srv55920725:83 , srv588690309:83 , srv686888876:83 , srv842631842:83 , srv1418221971:84 , srv1469859447:84 , srv1648103647:84 , srv183408682:84 , srv192352895:84 , srv2013722430:84 , srv257392250:84 , srv427552121:84 , srv798903960:84 , srv1211362611:85 , srv1353811044:85 , srv1433773553:85 , srv1527648277:85 , srv1728320641:85 , srv1903485773:85 , srv258877909:85 , srv293936383:85 , srv611178502:85 , srv704216461:85 , srv736301695:85 , srv1239750324:86 , srv1247109217:86 , srv127953046:86 , srv1566878416:86 , srv1575895823:86 , srv162469518:86 , srv181483074:86 , srv1887536122:86 , srv1894862012:86 , srv2082147938:86 , srv434655485:86 , srv537605653:86 , srv56402784:86 , srv587729608:86 , srv6347447:86 , srv716371335:86 , srv7435289:86 , srv75802784:86 , srv775431397:86 , srv103092926:87 , srv1063391143:87 , srv1098263132:87 , srv1153931253:87 , srv1272116287:87 , srv133202840:87 , srv1483826960:87 , srv1752767683:87 , srv1855758158:87 , srv1954331564:87 , srv2087923926:87 , srv2134760103:87 , srv241069586:87 , srv34760423:87 , srv366913283:87 , srv645100469:87 , srv68995494:87 , srv944023371:87 , srv952957057:87 , srv1309644784:88 , srv1601968701:88 , srv1847714149:88 , srv459211219:88 , srv597655144:88 , srv611469071:88 , srv652003344:88 , srv856944210:88 , srv1304286216:89 , srv2134612531:89 , srv415533951:89 , srv634903216:89 , srv1592801702:90 , srv2094984088:90 , srv388051609:90 , srv1655307662:91 , srv1951630017:92 , srv1230002842:546 } 2023-12-16 10:54:20,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 10:54:20,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv686888876=86, srv1439246241=22, srv181483074=36, srv2082147938=50, srv587729608=77, srv588690309=78, srv1422175831=20, srv1903485773=43, srv127953046=13, srv1304286216=15, srv192352895=45, srv68995494=87, srv1239760934=10, srv1433773553=21, srv1752767683=35, srv851727095=96, srv611469071=81, srv75802784=92, srv944023371=98, srv1153931253=5, srv7435289=91, srv1295993964=14, srv388051609=65, srv1353811044=18, srv1884251959=40, srv1483826960=24, srv1575895823=28, srv6347447=82, srv1951630017=46, srv1230002842=8, srv736301695=90, srv1272116287=12, srv44668621=70, srv162469518=31, srv427552121=67, srv257392250=58, srv293936383=60, srv2134612531=54, srv1917939982=44, srv2120927230=53, srv1592801702=29, srv2081021998=49, srv428871281=68, srv1222492841=7, srv1655307662=33, srv1079418042=2, srv1728320641=34, srv241069586=56, srv856944210=97, srv1247109217=11, srv2134760103=55, srv1239750324=9, srv1418221971=19, srv842631842=95, srv471291045=72, srv597655144=79, srv1847714149=38, srv798903960=94, srv334739251=61, srv1887536122=41, srv2094984088=52, srv704216461=88, srv2087923926=51, srv634903216=83, srv1469859447=23, srv133202840=17, srv1098263132=3, srv1855758158=39, srv2013722430=48, srv1211362611=6, srv366913283=63, srv1309644784=16, srv34760423=62, srv1063391143=1, srv508523515=73, srv1132476308=4, srv952957057=99, srv1527648277=26, srv1503311047=25, srv103092926=0, srv258877909=59, srv370716956=64, srv415533951=66, srv775431397=93, srv537605653=74, srv55920725=75, srv1566878416=27, srv645100469=84, srv459211219=71, srv716371335=89, srv652003344=85, srv1648103647=32, srv434655485=69, srv56402784=76, srv1894862012=42, srv256926400=57, srv1601968701=30, srv183408682=37, srv1954331564=47, srv611178502=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 10:54:20,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 10:54:20,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 10:54:20,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 10:54:20,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 10:54:20,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 10:54:20,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-16 10:54:20,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-16 10:54:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-16 10:54:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-16 10:54:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-16 10:54:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-16 10:54:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-16 10:54:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-16 10:54:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-16 10:54:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-16 10:54:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-16 10:54:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-16 10:54:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-16 10:54:20,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-16 10:54:20,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-16 10:54:20,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-16 10:54:20,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-16 10:54:20,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-16 10:54:20,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-16 10:54:20,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-16 10:54:20,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-16 10:54:20,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-16 10:54:20,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-16 10:54:20,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-16 10:54:20,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-16 10:54:20,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-16 10:54:20,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-16 10:54:20,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-16 10:54:20,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-16 10:54:20,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-16 10:54:20,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-16 10:54:20,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-16 10:54:20,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-16 10:54:20,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-16 10:54:20,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-16 10:54:20,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-16 10:54:20,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-16 10:54:20,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-16 10:54:20,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-16 10:54:20,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-16 10:54:20,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-16 10:54:20,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-16 10:54:20,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-16 10:54:20,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-16 10:54:20,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-16 10:54:20,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-16 10:54:20,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-16 10:54:20,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-16 10:54:20,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-16 10:54:20,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-16 10:54:20,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-16 10:54:20,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-16 10:54:20,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-16 10:54:20,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-16 10:54:20,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-16 10:54:20,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-16 10:54:20,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-16 10:54:20,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-16 10:54:20,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-16 10:54:20,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-16 10:54:20,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-16 10:54:20,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-16 10:54:20,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-16 10:54:20,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-16 10:54:20,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-16 10:54:20,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-16 10:54:20,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-16 10:54:20,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-16 10:54:20,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-16 10:54:21,104 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021376732884530333); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033618310440256566, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021376732884530333); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042583333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:54:21,104 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-16 10:54:21,104 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01481625995762716, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021376732884530333); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033618310440256566, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021376732884530333); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042583333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-16 10:54:21,438 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 434 ms to try 5000 different iterations. Found a solution that moves 980 regions; Going from a computed imbalance of 0.01481625995762716 to a new imbalance of 6.787679613139783E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006098088358750311); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007741618689438271); MoveCostFunction : (multiplier=7.0, imbalance=0.10888888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006098088358750311); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:54:21,440 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1247109217:82 , srv1433773553:82 , srv1951630017:82 , srv537605653:82 , srv103092926:83 , srv1752767683:83 , srv2094984088:83 , srv2134612531:83 , srv645100469:83 , srv1272116287:84 , srv1295993964:84 , srv1439246241:84 , srv1601968701:84 , srv1728320641:84 , srv1855758158:84 , srv1954331564:84 , srv2081021998:84 , srv2134760103:84 , srv241069586:84 , srv256926400:84 , srv366913283:84 , srv471291045:84 , srv597655144:84 , srv611178502:84 , srv6347447:84 , srv652003344:84 , srv75802784:84 , srv952957057:84 , srv1098263132:85 , srv1211362611:85 , srv1309644784:85 , srv1422175831:85 , srv1469859447:85 , srv162469518:85 , srv1655307662:85 , srv1847714149:85 , srv1894862012:85 , srv2082147938:85 , srv257392250:85 , srv388051609:85 , srv415533951:85 , srv427552121:85 , srv55920725:85 , srv588690309:85 , srv611469071:85 , srv634903216:85 , srv686888876:85 , srv68995494:85 , srv704216461:85 , srv736301695:85 , srv7435289:85 , srv1063391143:86 , srv1079418042:86 , srv1132476308:86 , srv1239750324:86 , srv127953046:86 , srv133202840:86 , srv1418221971:86 , srv1483826960:86 , srv1527648277:86 , srv1592801702:86 , srv1648103647:86 , srv181483074:86 , srv1884251959:86 , srv1887536122:86 , srv1917939982:86 , srv192352895:86 , srv2087923926:86 , srv293936383:86 , srv370716956:86 , srv434655485:86 , srv44668621:86 , srv56402784:86 , srv587729608:86 , srv716371335:86 , srv798903960:86 , srv1153931253:87 , srv1222492841:87 , srv1239760934:87 , srv1304286216:87 , srv1566878416:87 , srv1575895823:87 , srv183408682:87 , srv1903485773:87 , srv2013722430:87 , srv2120927230:87 , srv334739251:87 , srv34760423:87 , srv428871281:87 , srv775431397:87 , srv842631842:87 , srv856944210:87 , srv944023371:87 , srv1353811044:88 , srv1503311047:88 , srv258877909:88 , srv459211219:88 , srv851727095:88 , srv508523515:89 , srv1230002842:548 } 2023-12-16 10:54:21,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 10:54:21,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv686888876=86, srv1439246241=22, srv181483074=36, srv2082147938=50, srv587729608=77, srv588690309=78, srv1422175831=20, srv1903485773=43, srv127953046=13, srv1304286216=15, srv192352895=45, srv68995494=87, srv1239760934=10, srv1433773553=21, srv1752767683=35, srv851727095=96, srv611469071=81, srv75802784=92, srv944023371=98, srv1153931253=5, srv7435289=91, srv1295993964=14, srv388051609=65, srv1353811044=18, srv1884251959=40, srv1483826960=24, srv1575895823=28, srv6347447=82, srv1951630017=46, srv1230002842=8, srv736301695=90, srv1272116287=12, srv44668621=70, srv162469518=31, srv427552121=67, srv257392250=58, srv293936383=60, srv2134612531=54, srv1917939982=44, srv2120927230=53, srv1592801702=29, srv2081021998=49, srv428871281=68, srv1222492841=7, srv1655307662=33, srv1079418042=2, srv1728320641=34, srv241069586=56, srv856944210=97, srv1247109217=11, srv2134760103=55, srv1239750324=9, srv1418221971=19, srv842631842=95, srv471291045=72, srv597655144=79, srv1847714149=38, srv798903960=94, srv334739251=61, srv1887536122=41, srv2094984088=52, srv704216461=88, srv2087923926=51, srv634903216=83, srv1469859447=23, srv133202840=17, srv1098263132=3, srv1855758158=39, srv2013722430=48, srv1211362611=6, srv366913283=63, srv1309644784=16, srv34760423=62, srv1063391143=1, srv508523515=73, srv1132476308=4, srv952957057=99, srv1527648277=26, srv1503311047=25, srv103092926=0, srv258877909=59, srv370716956=64, srv415533951=66, srv775431397=93, srv537605653=74, srv55920725=75, srv1566878416=27, srv645100469=84, srv459211219=71, srv716371335=89, srv652003344=85, srv1648103647=32, srv434655485=69, srv56402784=76, srv1894862012=42, srv256926400=57, srv1601968701=30, srv183408682=37, srv1954331564=47, srv611178502=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 10:54:21,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 10:54:21,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 10:54:21,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 10:54:21,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 10:54:21,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 10:54:21,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 10:54:21,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 10:54:21,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 10:54:21,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 10:54:21,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 10:54:21,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 10:54:21,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 10:54:21,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 10:54:21,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 10:54:21,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 10:54:21,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 10:54:21,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 10:54:21,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 10:54:21,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 10:54:21,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 10:54:21,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 10:54:21,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 10:54:21,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 10:54:21,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 10:54:21,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 10:54:21,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 10:54:21,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 10:54:21,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 10:54:21,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 10:54:21,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 10:54:21,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-16 10:54:21,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-16 10:54:21,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-16 10:54:21,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-16 10:54:21,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-16 10:54:21,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-16 10:54:21,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-16 10:54:21,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-16 10:54:21,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-16 10:54:21,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-16 10:54:21,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-16 10:54:21,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-16 10:54:21,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-16 10:54:21,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-16 10:54:21,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-16 10:54:21,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-16 10:54:21,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-16 10:54:21,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-16 10:54:21,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-16 10:54:21,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-16 10:54:21,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-16 10:54:21,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-16 10:54:21,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-16 10:54:21,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-16 10:54:21,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-16 10:54:21,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-16 10:54:21,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-16 10:54:21,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-16 10:54:21,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-16 10:54:21,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-16 10:54:21,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-16 10:54:21,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-16 10:54:21,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-16 10:54:21,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-16 10:54:21,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-16 10:54:21,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-16 10:54:21,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-16 10:54:21,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-16 10:54:21,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-16 10:54:21,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-16 10:54:21,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-16 10:54:21,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-16 10:54:21,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-16 10:54:21,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-16 10:54:21,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-16 10:54:21,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-16 10:54:21,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-16 10:54:21,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-16 10:54:21,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-16 10:54:21,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-16 10:54:21,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-16 10:54:21,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-16 10:54:21,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-16 10:54:21,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-16 10:54:21,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-16 10:54:21,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-16 10:54:21,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-16 10:54:21,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-16 10:54:21,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-16 10:54:21,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-16 10:54:21,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-16 10:54:21,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-16 10:54:21,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-16 10:54:21,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-16 10:54:21,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-16 10:54:21,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-16 10:54:21,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-16 10:54:21,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-16 10:54:21,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-16 10:54:21,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-16 10:54:21,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-16 10:54:21,577 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006098088358750311); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007741618689438271); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006098088358750311); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:54:21,578 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-16 10:54:21,578 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.719058546472317E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006098088358750311); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007741618689438271); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006098088358750311); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-16 10:54:21,795 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 312 ms to try 5000 different iterations. Found a solution that moves 243 regions; Going from a computed imbalance of 6.719058546472317E-4 to a new imbalance of 3.4602282473669806E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.027, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:54:21,796 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2013722430:82 , srv1575895823:83 , srv183408682:83 , srv1884251959:83 , srv258877909:83 , srv459211219:83 , srv1063391143:84 , srv1079418042:84 , srv1098263132:84 , srv1239760934:84 , srv1272116287:84 , srv127953046:84 , srv1304286216:84 , srv1309644784:84 , srv133202840:84 , srv1353811044:84 , srv1433773553:84 , srv1483826960:84 , srv1503311047:84 , srv1527648277:84 , srv1566878416:84 , srv1592801702:84 , srv1648103647:84 , srv1855758158:84 , srv1951630017:84 , srv2081021998:84 , srv2087923926:84 , srv241069586:84 , srv34760423:84 , srv366913283:84 , srv434655485:84 , srv56402784:84 , srv587729608:84 , srv6347447:84 , srv686888876:84 , srv716371335:84 , srv775431397:84 , srv798903960:84 , srv851727095:84 , srv952957057:84 , srv103092926:85 , srv1132476308:85 , srv1153931253:85 , srv1211362611:85 , srv1222492841:85 , srv1239750324:85 , srv1295993964:85 , srv1418221971:85 , srv1422175831:85 , srv1439246241:85 , srv1469859447:85 , srv1601968701:85 , srv162469518:85 , srv1655307662:85 , srv1752767683:85 , srv181483074:85 , srv1847714149:85 , srv1887536122:85 , srv1894862012:85 , srv1903485773:85 , srv1917939982:85 , srv192352895:85 , srv1954331564:85 , srv2082147938:85 , srv2094984088:85 , srv2120927230:85 , srv2134612531:85 , srv2134760103:85 , srv256926400:85 , srv257392250:85 , srv293936383:85 , srv334739251:85 , srv370716956:85 , srv388051609:85 , srv415533951:85 , srv427552121:85 , srv428871281:85 , srv44668621:85 , srv471291045:85 , srv508523515:85 , srv537605653:85 , srv55920725:85 , srv588690309:85 , srv597655144:85 , srv611178502:85 , srv611469071:85 , srv634903216:85 , srv645100469:85 , srv652003344:85 , srv68995494:85 , srv704216461:85 , srv736301695:85 , srv7435289:85 , srv75802784:85 , srv842631842:85 , srv856944210:85 , srv944023371:85 , srv1247109217:86 , srv1728320641:86 , srv1230002842:630 } 2023-12-16 10:54:21,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 10:54:21,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv686888876=86, srv1439246241=22, srv181483074=36, srv2082147938=50, srv587729608=77, srv588690309=78, srv1422175831=20, srv1903485773=43, srv127953046=13, srv1304286216=15, srv192352895=45, srv68995494=87, srv1239760934=10, srv1433773553=21, srv1752767683=35, srv851727095=96, srv611469071=81, srv75802784=92, srv944023371=98, srv1153931253=5, srv7435289=91, srv1295993964=14, srv388051609=65, srv1353811044=18, srv1884251959=40, srv1483826960=24, srv1575895823=28, srv6347447=82, srv1951630017=46, srv1230002842=8, srv736301695=90, srv1272116287=12, srv44668621=70, srv162469518=31, srv427552121=67, srv257392250=58, srv293936383=60, srv2134612531=54, srv1917939982=44, srv2120927230=53, srv1592801702=29, srv2081021998=49, srv428871281=68, srv1222492841=7, srv1655307662=33, srv1079418042=2, srv1728320641=34, srv241069586=56, srv856944210=97, srv1247109217=11, srv2134760103=55, srv1239750324=9, srv1418221971=19, srv842631842=95, srv471291045=72, srv597655144=79, srv1847714149=38, srv798903960=94, srv334739251=61, srv1887536122=41, srv2094984088=52, srv704216461=88, srv2087923926=51, srv634903216=83, srv1469859447=23, srv133202840=17, srv1098263132=3, srv1855758158=39, srv2013722430=48, srv1211362611=6, srv366913283=63, srv1309644784=16, srv34760423=62, srv1063391143=1, srv508523515=73, srv1132476308=4, srv952957057=99, srv1527648277=26, srv1503311047=25, srv103092926=0, srv258877909=59, srv370716956=64, srv415533951=66, srv775431397=93, srv537605653=74, srv55920725=75, srv1566878416=27, srv645100469=84, srv459211219=71, srv716371335=89, srv652003344=85, srv1648103647=32, srv434655485=69, srv56402784=76, srv1894862012=42, srv256926400=57, srv1601968701=30, srv183408682=37, srv1954331564=47, srv611178502=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 10:54:21,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 10:54:21,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 10:54:21,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 10:54:21,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 10:54:21,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 10:54:21,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 10:54:21,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 10:54:21,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-16 10:54:21,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-16 10:54:21,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-16 10:54:21,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-16 10:54:21,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-16 10:54:21,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-16 10:54:21,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-16 10:54:21,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-16 10:54:21,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-16 10:54:21,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-16 10:54:21,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-16 10:54:21,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-16 10:54:21,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-16 10:54:21,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-16 10:54:21,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-16 10:54:21,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-16 10:54:21,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-16 10:54:21,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-16 10:54:21,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-16 10:54:21,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-16 10:54:21,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-16 10:54:21,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-16 10:54:21,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-16 10:54:21,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-16 10:54:21,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-16 10:54:21,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-16 10:54:21,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-16 10:54:21,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-16 10:54:21,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-16 10:54:21,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-16 10:54:21,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-16 10:54:21,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-16 10:54:21,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-16 10:54:21,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-16 10:54:21,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-16 10:54:21,923 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:54:21,923 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-16 10:54:21,923 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.290076010630302E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-16 10:54:22,128 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 302 ms to try 5000 different iterations. Found a solution that moves 26 regions; Going from a computed imbalance of 3.290076010630302E-5 to a new imbalance of 6.926124944990534E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0028888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:54:22,129 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1728320641:83 , srv1903485773:83 , srv508523515:83 , srv634903216:83 , srv652003344:83 , srv856944210:83 , srv103092926:84 , srv1063391143:84 , srv1079418042:84 , srv1098263132:84 , srv1132476308:84 , srv1153931253:84 , srv1211362611:84 , srv1222492841:84 , srv1239750324:84 , srv1239760934:84 , srv1247109217:84 , srv1272116287:84 , srv127953046:84 , srv1295993964:84 , srv1304286216:84 , srv133202840:84 , srv1353811044:84 , srv1418221971:84 , srv1422175831:84 , srv1433773553:84 , srv1439246241:84 , srv1469859447:84 , srv1483826960:84 , srv1503311047:84 , srv1527648277:84 , srv1566878416:84 , srv1592801702:84 , srv1601968701:84 , srv162469518:84 , srv1648103647:84 , srv1655307662:84 , srv1752767683:84 , srv181483074:84 , srv1847714149:84 , srv1855758158:84 , srv1887536122:84 , srv1894862012:84 , srv1917939982:84 , srv192352895:84 , srv1951630017:84 , srv1954331564:84 , srv2013722430:84 , srv2081021998:84 , srv2082147938:84 , srv2087923926:84 , srv2094984088:84 , srv2120927230:84 , srv2134612531:84 , srv2134760103:84 , srv256926400:84 , srv257392250:84 , srv258877909:84 , srv293936383:84 , srv334739251:84 , srv34760423:84 , srv366913283:84 , srv370716956:84 , srv388051609:84 , srv415533951:84 , srv427552121:84 , srv428871281:84 , srv434655485:84 , srv44668621:84 , srv459211219:84 , srv471291045:84 , srv537605653:84 , srv55920725:84 , srv587729608:84 , srv588690309:84 , srv597655144:84 , srv611178502:84 , srv611469071:84 , srv6347447:84 , srv645100469:84 , srv686888876:84 , srv68995494:84 , srv704216461:84 , srv716371335:84 , srv736301695:84 , srv7435289:84 , srv75802784:84 , srv775431397:84 , srv798903960:84 , srv842631842:84 , srv851727095:84 , srv944023371:84 , srv952957057:84 , srv1309644784:85 , srv1575895823:85 , srv183408682:85 , srv1884251959:85 , srv241069586:85 , srv56402784:85 , srv1230002842:684 } 2023-12-16 10:54:22,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 10:54:22,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv686888876=86, srv1439246241=22, srv181483074=36, srv2082147938=50, srv587729608=77, srv588690309=78, srv1422175831=20, srv1903485773=43, srv127953046=13, srv1304286216=15, srv192352895=45, srv68995494=87, srv1239760934=10, srv1433773553=21, srv1752767683=35, srv851727095=96, srv611469071=81, srv75802784=92, srv944023371=98, srv1153931253=5, srv7435289=91, srv1295993964=14, srv388051609=65, srv1353811044=18, srv1884251959=40, srv1483826960=24, srv1575895823=28, srv6347447=82, srv1951630017=46, srv1230002842=8, srv736301695=90, srv1272116287=12, srv44668621=70, srv162469518=31, srv427552121=67, srv257392250=58, srv293936383=60, srv2134612531=54, srv1917939982=44, srv2120927230=53, srv1592801702=29, srv2081021998=49, srv428871281=68, srv1222492841=7, srv1655307662=33, srv1079418042=2, srv1728320641=34, srv241069586=56, srv856944210=97, srv1247109217=11, srv2134760103=55, srv1239750324=9, srv1418221971=19, srv842631842=95, srv471291045=72, srv597655144=79, srv1847714149=38, srv798903960=94, srv334739251=61, srv1887536122=41, srv2094984088=52, srv704216461=88, srv2087923926=51, srv634903216=83, srv1469859447=23, srv133202840=17, srv1098263132=3, srv1855758158=39, srv2013722430=48, srv1211362611=6, srv366913283=63, srv1309644784=16, srv34760423=62, srv1063391143=1, srv508523515=73, srv1132476308=4, srv952957057=99, srv1527648277=26, srv1503311047=25, srv103092926=0, srv258877909=59, srv370716956=64, srv415533951=66, srv775431397=93, srv537605653=74, srv55920725=75, srv1566878416=27, srv645100469=84, srv459211219=71, srv716371335=89, srv652003344=85, srv1648103647=32, srv434655485=69, srv56402784=76, srv1894862012=42, srv256926400=57, srv1601968701=30, srv183408682=37, srv1954331564=47, srv611178502=80} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 10:54:22,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 10:54:22,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-16 10:54:22,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-16 10:54:22,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-16 10:54:22,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-16 10:54:22,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-16 10:54:22,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-16 10:54:22,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-16 10:54:22,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-16 10:54:22,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-16 10:54:22,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-16 10:54:22,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-16 10:54:22,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-16 10:54:22,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-16 10:54:22,262 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:54:22,262 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1728320641:83 , srv1903485773:83 , srv508523515:83 , srv634903216:83 , srv652003344:83 , srv856944210:83 , srv103092926:84 , srv1063391143:84 , srv1079418042:84 , srv1098263132:84 , srv1132476308:84 , srv1153931253:84 , srv1211362611:84 , srv1222492841:84 , srv1239750324:84 , srv1239760934:84 , srv1247109217:84 , srv1272116287:84 , srv127953046:84 , srv1295993964:84 , srv1304286216:84 , srv133202840:84 , srv1353811044:84 , srv1418221971:84 , srv1422175831:84 , srv1433773553:84 , srv1439246241:84 , srv1469859447:84 , srv1483826960:84 , srv1503311047:84 , srv1527648277:84 , srv1566878416:84 , srv1592801702:84 , srv1601968701:84 , srv162469518:84 , srv1648103647:84 , srv1655307662:84 , srv1752767683:84 , srv181483074:84 , srv1847714149:84 , srv1855758158:84 , srv1887536122:84 , srv1894862012:84 , srv1917939982:84 , srv192352895:84 , srv1951630017:84 , srv1954331564:84 , srv2013722430:84 , srv2081021998:84 , srv2082147938:84 , srv2087923926:84 , srv2094984088:84 , srv2120927230:84 , srv2134612531:84 , srv2134760103:84 , srv256926400:84 , srv257392250:84 , srv258877909:84 , srv293936383:84 , srv334739251:84 , srv34760423:84 , srv366913283:84 , srv370716956:84 , srv388051609:84 , srv415533951:84 , srv427552121:84 , srv428871281:84 , srv434655485:84 , srv44668621:84 , srv459211219:84 , srv471291045:84 , srv537605653:84 , srv55920725:84 , srv587729608:84 , srv588690309:84 , srv597655144:84 , srv611178502:84 , srv611469071:84 , srv6347447:84 , srv645100469:84 , srv686888876:84 , srv68995494:84 , srv704216461:84 , srv716371335:84 , srv736301695:84 , srv7435289:84 , srv75802784:84 , srv775431397:84 , srv798903960:84 , srv842631842:84 , srv851727095:84 , srv944023371:84 , srv952957057:84 , srv1309644784:85 , srv1575895823:85 , srv183408682:85 , srv1884251959:85 , srv241069586:85 , srv56402784:85 , srv1230002842:684 } 2023-12-16 10:54:22,294 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=327 (was 321) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=6727 (was 7360) 2023-12-16 10:54:22,301 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=327, ProcessCount=168, AvailableMemoryMB=6726 2023-12-16 10:54:22,302 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-16 10:54:22,302 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-16 10:54:22,313 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1006148408:84 , srv1077933851:84 , srv1390042060:84 , srv1457660351:84 , srv1537533452:84 , srv1540175129:84 , srv1600624016:84 , srv1616088913:84 , srv1642243429:84 , srv1693068814:84 , srv1699810441:84 , srv1737980335:84 , srv1882231015:84 , srv1900272117:84 , srv1962500146:84 , srv2075153966:84 , srv268115663:84 , srv314306094:84 , srv378773344:84 , srv380246853:84 , srv584481344:84 , srv618358413:84 , srv624818655:84 , srv656630422:84 , srv764559851:84 , srv795074761:84 , srv80120724:84 , srv850842570:84 , srv970817377:84 , srv687737354:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-16 10:54:22,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 10:54:22,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1693068814=9, srv80120724=27, srv1882231015=12, srv1900272117=13, srv795074761=26, srv1616088913=7, srv850842570=28, srv1390042060=2, srv970817377=29, srv764559851=25, srv1077933851=1, srv584481344=20, srv687737354=24, srv1537533452=4, srv1642243429=8, srv380246853=19, srv1006148408=0, srv1540175129=5, srv1737980335=11, srv378773344=18, srv2075153966=15, srv1699810441=10, srv1962500146=14, srv656630422=23, srv624818655=22, srv314306094=17, srv618358413=21, srv268115663=16, srv1457660351=3, srv1600624016=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 10:54:22,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 10:54:22,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 10:54:22,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 10:54:22,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 10:54:22,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 10:54:22,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 10:54:22,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 10:54:22,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 10:54:22,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-16 10:54:22,342 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0984695153875161, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:54:22,342 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991250331749194, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0984695153875161, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-16 10:56:22,330 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6421799 different iterations. Found a solution that moves 1854 regions; Going from a computed imbalance of 0.991250331749194 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:56:22,336 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1006148408:90 , srv1077933851:90 , srv1390042060:90 , srv1457660351:90 , srv1537533452:90 , srv1540175129:90 , srv1600624016:90 , srv1616088913:90 , srv1642243429:90 , srv1693068814:90 , srv1699810441:90 , srv1737980335:90 , srv1882231015:90 , srv1900272117:90 , srv1962500146:90 , srv2075153966:90 , srv268115663:90 , srv314306094:90 , srv378773344:90 , srv380246853:90 , srv584481344:90 , srv618358413:90 , srv624818655:90 , srv656630422:90 , srv687737354:90 , srv764559851:90 , srv795074761:90 , srv80120724:90 , srv850842570:90 , srv970817377:90 } 2023-12-16 10:56:22,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-16 10:56:22,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1693068814=9, srv80120724=27, srv1882231015=12, srv1900272117=13, srv795074761=26, srv1616088913=7, srv850842570=28, srv1390042060=2, srv970817377=29, srv764559851=25, srv1077933851=1, srv584481344=20, srv687737354=24, srv1537533452=4, srv1642243429=8, srv380246853=19, srv1006148408=0, srv1540175129=5, srv1737980335=11, srv378773344=18, srv2075153966=15, srv1699810441=10, srv1962500146=14, srv656630422=23, srv624818655=22, srv314306094=17, srv618358413=21, srv268115663=16, srv1457660351=3, srv1600624016=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-16 10:56:22,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-16 10:56:22,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-16 10:56:22,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-16 10:56:22,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-16 10:56:22,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-16 10:56:22,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-16 10:56:22,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-16 10:56:22,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-16 10:56:22,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-16 10:56:22,353 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-16 10:56:22,353 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1006148408:90 , srv1077933851:90 , srv1390042060:90 , srv1457660351:90 , srv1537533452:90 , srv1540175129:90 , srv1600624016:90 , srv1616088913:90 , srv1642243429:90 , srv1693068814:90 , srv1699810441:90 , srv1737980335:90 , srv1882231015:90 , srv1900272117:90 , srv1962500146:90 , srv2075153966:90 , srv268115663:90 , srv314306094:90 , srv378773344:90 , srv380246853:90 , srv584481344:90 , srv618358413:90 , srv624818655:90 , srv656630422:90 , srv687737354:90 , srv764559851:90 , srv795074761:90 , srv80120724:90 , srv850842570:90 , srv970817377:90 } 2023-12-16 10:56:22,366 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 327), ProcessCount=167 (was 168), AvailableMemoryMB=5785 (was 6726)