2024-03-19 09:53:24,435 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-03-19 09:53:24,729 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-03-19 09:53:24,830 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-03-19 09:53:24,891 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-03-19 09:53:24,898 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-03-19 09:53:24,914 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=248, ProcessCount=170, AvailableMemoryMB=4044 2024-03-19 09:53:24,916 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-03-19 09:53:24,916 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-03-19 09:53:25,025 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1034641065:84 , srv1059631235:84 , srv1067358804:84 , srv1078276412:84 , srv1130343184:84 , srv1242410403:84 , srv1246496094:84 , srv1281673890:84 , srv1293507742:84 , srv1306074788:84 , srv1311668677:84 , srv1312635970:84 , srv1337035360:84 , srv1367984024:84 , srv1440941716:84 , srv1512038533:84 , srv1513235373:84 , srv1519314644:84 , srv1549295911:84 , srv1550559515:84 , srv1625445930:84 , srv163817489:84 , srv1638298839:84 , srv1644793900:84 , srv1649399700:84 , srv1659054976:84 , srv1668284265:84 , srv1693985027:84 , srv1747400141:84 , srv1754514288:84 , srv176317320:84 , srv176538357:84 , srv1774238949:84 , srv1805335123:84 , srv1816073632:84 , srv1843171822:84 , srv1906964914:84 , srv1917867363:84 , srv1940175115:84 , srv1958810416:84 , srv1963283101:84 , srv19953607:84 , srv2001543385:84 , srv2005405000:84 , srv2039290169:84 , srv2099046634:84 , srv2107638748:84 , srv250877291:84 , srv268346097:84 , srv273900190:84 , srv306008607:84 , srv344100082:84 , srv346695473:84 , srv358881020:84 , srv374901804:84 , srv378055747:84 , srv400302884:84 , srv418320518:84 , srv428984924:84 , srv429225318:84 , srv444245511:84 , srv472507203:84 , srv483452625:84 , srv490713531:84 , srv521439280:84 , srv593882282:84 , srv602092061:84 , srv604626877:84 , srv605149346:84 , srv622923211:84 , srv637914902:84 , srv638367536:84 , srv648718789:84 , srv658317019:84 , srv682776345:84 , srv689068841:84 , srv700159450:84 , srv702361237:84 , srv719589491:84 , srv731118474:84 , srv736281688:84 , srv764526262:84 , srv771698455:84 , srv78414130:84 , srv791774291:84 , srv802360655:84 , srv809846139:84 , srv835880913:84 , srv872230395:84 , srv894681765:84 , srv911611954:84 , srv913586963:84 , srv921682066:84 , srv940590657:84 , srv942291463:84 , srv949993510:84 , srv966137735:84 , srv997073386:84 , srv997665026:84 , srv277492397:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-03-19 09:53:25,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-19 09:53:25,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1649399700=24, srv700159450=77, srv1659054976=25, srv1774238949=32, srv1963283101=40, srv2001543385=42, srv648718789=73, srv802360655=86, srv997665026=99, srv966137735=97, srv658317019=74, srv472507203=62, srv268346097=48, srv1906964914=36, srv1512038533=15, srv1130343184=4, srv483452625=63, srv940590657=94, srv428984924=59, srv638367536=72, srv1034641065=0, srv1337035360=12, srv400302884=57, srv764526262=82, srv949993510=96, srv1917867363=37, srv1312635970=11, srv605149346=69, srv1059631235=1, srv622923211=70, srv1440941716=14, srv1550559515=19, srv791774291=85, srv771698455=83, srv1242410403=5, srv682776345=75, srv1816073632=34, srv490713531=64, srv835880913=88, srv1958810416=39, srv429225318=60, srv1281673890=7, srv2099046634=45, srv176317320=30, srv736281688=81, srv1513235373=16, srv418320518=58, srv19953607=41, srv921682066=93, srv273900190=49, srv731118474=80, srv250877291=47, srv374901804=55, srv163817489=21, srv1549295911=18, srv1638298839=22, srv689068841=76, srv911611954=91, srv1940175115=38, srv894681765=90, srv1754514288=29, srv593882282=66, srv1293507742=8, srv1747400141=28, srv2039290169=44, srv1668284265=26, srv1246496094=6, srv1519314644=17, srv2107638748=46, srv378055747=56, srv702361237=78, srv444245511=61, srv637914902=71, srv277492397=50, srv306008607=51, srv521439280=65, srv809846139=87, srv1306074788=9, srv358881020=54, srv346695473=53, srv1367984024=13, srv344100082=52, srv1644793900=23, srv1843171822=35, srv1625445930=20, srv176538357=31, srv1805335123=33, srv872230395=89, srv1311668677=10, srv602092061=67, srv1693985027=27, srv604626877=68, srv1078276412=3, srv719589491=79, srv78414130=84, srv913586963=92, srv997073386=98, srv1067358804=2, srv942291463=95, srv2005405000=43} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-19 09:53:25,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-19 09:53:25,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-19 09:53:25,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-19 09:53:25,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-19 09:53:25,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-19 09:53:25,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-19 09:53:25,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-19 09:53:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-19 09:53:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-19 09:53:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-19 09:53:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-19 09:53:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-19 09:53:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-19 09:53:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-19 09:53:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-19 09:53:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-19 09:53:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-19 09:53:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-19 09:53:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-19 09:53:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-19 09:53:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-19 09:53:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-19 09:53:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-19 09:53:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-19 09:53:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-19 09:53:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-19 09:53:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-19 09:53:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-19 09:53:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-19 09:53:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-19 09:53:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-03-19 09:53:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-03-19 09:53:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-03-19 09:53:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-03-19 09:53:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-03-19 09:53:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-03-19 09:53:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-03-19 09:53:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-03-19 09:53:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-03-19 09:53:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-03-19 09:53:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-03-19 09:53:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-03-19 09:53:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-03-19 09:53:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-03-19 09:53:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-03-19 09:53:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-03-19 09:53:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-03-19 09:53:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-03-19 09:53:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-03-19 09:53:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-03-19 09:53:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-03-19 09:53:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-03-19 09:53:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-03-19 09:53:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-03-19 09:53:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-03-19 09:53:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-03-19 09:53:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-03-19 09:53:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-03-19 09:53:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-03-19 09:53:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-03-19 09:53:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-03-19 09:53:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-03-19 09:53:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-03-19 09:53:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-03-19 09:53:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-03-19 09:53:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-03-19 09:53:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-03-19 09:53:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-03-19 09:53:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-03-19 09:53:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-03-19 09:53:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-03-19 09:53:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-03-19 09:53:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-03-19 09:53:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-03-19 09:53:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-03-19 09:53:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-03-19 09:53:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-03-19 09:53:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-03-19 09:53:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-03-19 09:53:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-03-19 09:53:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-03-19 09:53:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-03-19 09:53:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-03-19 09:53:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-03-19 09:53:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-03-19 09:53:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-03-19 09:53:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-03-19 09:53:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-03-19 09:53:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-03-19 09:53:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-03-19 09:53:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-03-19 09:53:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-03-19 09:53:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-03-19 09:53:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-03-19 09:53:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-03-19 09:53:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-03-19 09:53:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-03-19 09:53:25,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-03-19 09:53:25,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-03-19 09:53:25,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-03-19 09:53:25,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-03-19 09:53:25,320 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:53:25,320 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-03-19 09:53:25,321 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-03-19 09:53:25,749 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 603 ms to try 5000 different iterations. Found a solution that moves 4676 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18653536950138455. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04250745406782316, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06015505591356461, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5195555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04250745406782316, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18191666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:53:25,771 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv378055747:82 , srv1034641065:83 , srv1067358804:83 , srv1311668677:83 , srv1337035360:83 , srv1550559515:83 , srv1747400141:83 , srv268346097:83 , srv835880913:83 , srv913586963:83 , srv966137735:83 , srv997665026:83 , srv1130343184:84 , srv1512038533:84 , srv1638298839:84 , srv2107638748:84 , srv658317019:84 , srv702361237:84 , srv1078276412:85 , srv1242410403:85 , srv1312635970:85 , srv1367984024:85 , srv1659054976:85 , srv1906964914:85 , srv1917867363:85 , srv1940175115:85 , srv2001543385:85 , srv344100082:85 , srv418320518:85 , srv490713531:85 , srv593882282:85 , srv602092061:85 , srv605149346:85 , srv637914902:85 , srv648718789:85 , srv700159450:85 , srv764526262:85 , srv791774291:85 , srv911611954:85 , srv921682066:85 , srv949993510:85 , srv1306074788:86 , srv1440941716:86 , srv1549295911:86 , srv1625445930:86 , srv1754514288:86 , srv1774238949:86 , srv1805335123:86 , srv1958810416:86 , srv19953607:86 , srv2005405000:86 , srv2039290169:86 , srv2099046634:86 , srv250877291:86 , srv306008607:86 , srv428984924:86 , srv429225318:86 , srv444245511:86 , srv472507203:86 , srv689068841:86 , srv771698455:86 , srv802360655:86 , srv997073386:86 , srv163817489:87 , srv1644793900:87 , srv1668284265:87 , srv1693985027:87 , srv176317320:87 , srv1843171822:87 , srv1963283101:87 , srv358881020:87 , srv374901804:87 , srv400302884:87 , srv604626877:87 , srv719589491:87 , srv731118474:87 , srv736281688:87 , srv78414130:87 , srv940590657:87 , srv1059631235:88 , srv1281673890:88 , srv1513235373:88 , srv1519314644:88 , srv1649399700:88 , srv176538357:88 , srv1816073632:88 , srv273900190:88 , srv521439280:88 , srv622923211:88 , srv942291463:88 , srv1293507742:89 , srv1246496094:90 , srv682776345:90 , srv346695473:91 , srv872230395:91 , srv638367536:92 , srv894681765:92 , srv809846139:94 , srv483452625:97 , srv277492397:468 } 2024-03-19 09:53:25,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-19 09:53:25,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1649399700=24, srv700159450=77, srv1659054976=25, srv1774238949=32, srv1963283101=40, srv2001543385=42, srv648718789=73, srv802360655=86, srv997665026=99, srv966137735=97, srv658317019=74, srv472507203=62, srv268346097=48, srv1906964914=36, srv1512038533=15, srv1130343184=4, srv483452625=63, srv940590657=94, srv428984924=59, srv638367536=72, srv1034641065=0, srv1337035360=12, srv400302884=57, srv764526262=82, srv949993510=96, srv1917867363=37, srv1312635970=11, srv605149346=69, srv1059631235=1, srv622923211=70, srv1440941716=14, srv1550559515=19, srv791774291=85, srv771698455=83, srv1242410403=5, srv682776345=75, srv1816073632=34, srv490713531=64, srv835880913=88, srv1958810416=39, srv429225318=60, srv1281673890=7, srv2099046634=45, srv176317320=30, srv736281688=81, srv1513235373=16, srv418320518=58, srv19953607=41, srv921682066=93, srv273900190=49, srv731118474=80, srv250877291=47, srv374901804=55, srv163817489=21, srv1549295911=18, srv1638298839=22, srv689068841=76, srv911611954=91, srv1940175115=38, srv894681765=90, srv1754514288=29, srv593882282=66, srv1293507742=8, srv1747400141=28, srv2039290169=44, srv1668284265=26, srv1246496094=6, srv1519314644=17, srv2107638748=46, srv378055747=56, srv702361237=78, srv444245511=61, srv637914902=71, srv277492397=50, srv306008607=51, srv521439280=65, srv809846139=87, srv1306074788=9, srv358881020=54, srv346695473=53, srv1367984024=13, srv344100082=52, srv1644793900=23, srv1843171822=35, srv1625445930=20, srv176538357=31, srv1805335123=33, srv872230395=89, srv1311668677=10, srv602092061=67, srv1693985027=27, srv604626877=68, srv1078276412=3, srv719589491=79, srv78414130=84, srv913586963=92, srv997073386=98, srv1067358804=2, srv942291463=95, srv2005405000=43} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-19 09:53:25,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-19 09:53:25,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-19 09:53:25,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-19 09:53:25,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-19 09:53:25,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-19 09:53:25,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-19 09:53:25,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-19 09:53:25,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-19 09:53:25,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-19 09:53:25,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-19 09:53:25,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-19 09:53:25,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-19 09:53:25,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-19 09:53:25,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-19 09:53:25,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-19 09:53:25,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-19 09:53:25,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-03-19 09:53:25,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-03-19 09:53:25,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-03-19 09:53:25,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-03-19 09:53:25,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-03-19 09:53:25,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-03-19 09:53:25,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-03-19 09:53:25,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-03-19 09:53:25,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-03-19 09:53:25,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-03-19 09:53:25,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-03-19 09:53:25,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-03-19 09:53:25,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-03-19 09:53:25,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-03-19 09:53:25,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-03-19 09:53:25,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-03-19 09:53:25,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-03-19 09:53:25,966 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04250745406782316, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06015505591356461, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04250745406782316, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18191666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:53:25,967 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-03-19 09:53:25,967 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18650262744957466, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04250745406782316, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06015505591356461, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04250745406782316, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18191666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-03-19 09:53:26,257 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 438 ms to try 5000 different iterations. Found a solution that moves 2680 regions; Going from a computed imbalance of 0.18650262744957466 to a new imbalance of 0.015500532907364934. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022611627461792343); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03680872389641014, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29777777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022611627461792343); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:53:26,266 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv483452625:77 , srv809846139:77 , srv894681765:80 , srv1246496094:81 , srv1293507742:81 , srv872230395:81 , srv163817489:82 , srv622923211:82 , srv1059631235:83 , srv1281673890:83 , srv1649399700:83 , srv1668284265:83 , srv176538357:83 , srv638367536:83 , srv682776345:83 , srv731118474:83 , srv942291463:83 , srv273900190:84 , srv306008607:84 , srv346695473:84 , srv358881020:84 , srv472507203:84 , srv521439280:84 , srv719589491:84 , srv791774291:84 , srv1306074788:85 , srv1440941716:85 , srv1519314644:85 , srv1693985027:85 , srv1754514288:85 , srv1816073632:85 , srv1843171822:85 , srv1906964914:85 , srv1963283101:85 , srv2039290169:85 , srv374901804:85 , srv418320518:85 , srv648718789:85 , srv736281688:85 , srv802360655:85 , srv1512038533:86 , srv1549295911:86 , srv1625445930:86 , srv1638298839:86 , srv1644793900:86 , srv1659054976:86 , srv176317320:86 , srv1774238949:86 , srv1805335123:86 , srv1917867363:86 , srv19953607:86 , srv250877291:86 , srv400302884:86 , srv429225318:86 , srv444245511:86 , srv602092061:86 , srv604626877:86 , srv78414130:86 , srv940590657:86 , srv949993510:86 , srv1513235373:87 , srv1940175115:87 , srv2001543385:87 , srv2005405000:87 , srv2107638748:87 , srv378055747:87 , srv428984924:87 , srv490713531:87 , srv593882282:87 , srv658317019:87 , srv702361237:87 , srv997073386:87 , srv1078276412:88 , srv1242410403:88 , srv1312635970:88 , srv1367984024:88 , srv1958810416:88 , srv605149346:88 , srv689068841:88 , srv764526262:88 , srv771698455:88 , srv911611954:88 , srv913586963:88 , srv921682066:88 , srv997665026:88 , srv1034641065:89 , srv1130343184:89 , srv1311668677:89 , srv1550559515:89 , srv1747400141:89 , srv268346097:89 , srv344100082:89 , srv637914902:89 , srv700159450:89 , srv835880913:89 , srv966137735:89 , srv1337035360:90 , srv2099046634:90 , srv1067358804:91 , srv277492397:507 } 2024-03-19 09:53:26,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-19 09:53:26,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1649399700=24, srv700159450=77, srv1659054976=25, srv1774238949=32, srv1963283101=40, srv2001543385=42, srv648718789=73, srv802360655=86, srv997665026=99, srv966137735=97, srv658317019=74, srv472507203=62, srv268346097=48, srv1906964914=36, srv1512038533=15, srv1130343184=4, srv483452625=63, srv940590657=94, srv428984924=59, srv638367536=72, srv1034641065=0, srv1337035360=12, srv400302884=57, srv764526262=82, srv949993510=96, srv1917867363=37, srv1312635970=11, srv605149346=69, srv1059631235=1, srv622923211=70, srv1440941716=14, srv1550559515=19, srv791774291=85, srv771698455=83, srv1242410403=5, srv682776345=75, srv1816073632=34, srv490713531=64, srv835880913=88, srv1958810416=39, srv429225318=60, srv1281673890=7, srv2099046634=45, srv176317320=30, srv736281688=81, srv1513235373=16, srv418320518=58, srv19953607=41, srv921682066=93, srv273900190=49, srv731118474=80, srv250877291=47, srv374901804=55, srv163817489=21, srv1549295911=18, srv1638298839=22, srv689068841=76, srv911611954=91, srv1940175115=38, srv894681765=90, srv1754514288=29, srv593882282=66, srv1293507742=8, srv1747400141=28, srv2039290169=44, srv1668284265=26, srv1246496094=6, srv1519314644=17, srv2107638748=46, srv378055747=56, srv702361237=78, srv444245511=61, srv637914902=71, srv277492397=50, srv306008607=51, srv521439280=65, srv809846139=87, srv1306074788=9, srv358881020=54, srv346695473=53, srv1367984024=13, srv344100082=52, srv1644793900=23, srv1843171822=35, srv1625445930=20, srv176538357=31, srv1805335123=33, srv872230395=89, srv1311668677=10, srv602092061=67, srv1693985027=27, srv604626877=68, srv1078276412=3, srv719589491=79, srv78414130=84, srv913586963=92, srv997073386=98, srv1067358804=2, srv942291463=95, srv2005405000=43} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-19 09:53:26,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-19 09:53:26,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-19 09:53:26,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-19 09:53:26,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-19 09:53:26,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-19 09:53:26,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-19 09:53:26,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-19 09:53:26,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-19 09:53:26,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-19 09:53:26,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-19 09:53:26,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-19 09:53:26,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-19 09:53:26,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-19 09:53:26,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-03-19 09:53:26,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-03-19 09:53:26,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-03-19 09:53:26,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-03-19 09:53:26,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-03-19 09:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-03-19 09:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-03-19 09:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-03-19 09:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-03-19 09:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-03-19 09:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-03-19 09:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-03-19 09:53:26,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-03-19 09:53:26,433 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022611627461792343); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03680872389641014, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022611627461792343); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:53:26,433 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-03-19 09:53:26,433 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015481767146276281, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022611627461792343); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03680872389641014, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022611627461792343); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-03-19 09:53:26,678 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 366 ms to try 5000 different iterations. Found a solution that moves 1017 regions; Going from a computed imbalance of 0.015481767146276281 to a new imbalance of 8.392366052909296E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0050929751922394705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007741618689438271); MoveCostFunction : (multiplier=7.0, imbalance=0.113, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0050929751922394705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:53:26,680 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2099046634:81 , srv1067358804:83 , srv346695473:83 , srv637914902:83 , srv638367536:83 , srv689068841:83 , srv1078276412:84 , srv1337035360:84 , srv1513235373:84 , srv1816073632:84 , srv1958810416:84 , srv344100082:84 , srv604626877:84 , srv682776345:84 , srv700159450:84 , srv764526262:84 , srv771698455:84 , srv911611954:84 , srv921682066:84 , srv1034641065:85 , srv1130343184:85 , srv1242410403:85 , srv1311668677:85 , srv1312635970:85 , srv1367984024:85 , srv1519314644:85 , srv1549295911:85 , srv1644793900:85 , srv1747400141:85 , srv176317320:85 , srv1805335123:85 , srv1843171822:85 , srv1963283101:85 , srv19953607:85 , srv268346097:85 , srv374901804:85 , srv400302884:85 , srv428984924:85 , srv429225318:85 , srv444245511:85 , srv483452625:85 , srv490713531:85 , srv593882282:85 , srv605149346:85 , srv78414130:85 , srv894681765:85 , srv940590657:85 , srv997073386:85 , srv1059631235:86 , srv1246496094:86 , srv1281673890:86 , srv1550559515:86 , srv1625445930:86 , srv1649399700:86 , srv1659054976:86 , srv1693985027:86 , srv1754514288:86 , srv1774238949:86 , srv1940175115:86 , srv2001543385:86 , srv2005405000:86 , srv250877291:86 , srv273900190:86 , srv521439280:86 , srv602092061:86 , srv658317019:86 , srv719589491:86 , srv736281688:86 , srv802360655:86 , srv809846139:86 , srv835880913:86 , srv872230395:86 , srv913586963:86 , srv966137735:86 , srv997665026:86 , srv1293507742:87 , srv1306074788:87 , srv1440941716:87 , srv1512038533:87 , srv1638298839:87 , srv1668284265:87 , srv176538357:87 , srv1917867363:87 , srv2039290169:87 , srv2107638748:87 , srv306008607:87 , srv358881020:87 , srv418320518:87 , srv472507203:87 , srv622923211:87 , srv702361237:87 , srv942291463:87 , srv163817489:88 , srv1906964914:88 , srv648718789:88 , srv731118474:88 , srv791774291:88 , srv949993510:88 , srv378055747:90 , srv277492397:528 } 2024-03-19 09:53:26,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-19 09:53:26,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1649399700=24, srv700159450=77, srv1659054976=25, srv1774238949=32, srv1963283101=40, srv2001543385=42, srv648718789=73, srv802360655=86, srv997665026=99, srv966137735=97, srv658317019=74, srv472507203=62, srv268346097=48, srv1906964914=36, srv1512038533=15, srv1130343184=4, srv483452625=63, srv940590657=94, srv428984924=59, srv638367536=72, srv1034641065=0, srv1337035360=12, srv400302884=57, srv764526262=82, srv949993510=96, srv1917867363=37, srv1312635970=11, srv605149346=69, srv1059631235=1, srv622923211=70, srv1440941716=14, srv1550559515=19, srv791774291=85, srv771698455=83, srv1242410403=5, srv682776345=75, srv1816073632=34, srv490713531=64, srv835880913=88, srv1958810416=39, srv429225318=60, srv1281673890=7, srv2099046634=45, srv176317320=30, srv736281688=81, srv1513235373=16, srv418320518=58, srv19953607=41, srv921682066=93, srv273900190=49, srv731118474=80, srv250877291=47, srv374901804=55, srv163817489=21, srv1549295911=18, srv1638298839=22, srv689068841=76, srv911611954=91, srv1940175115=38, srv894681765=90, srv1754514288=29, srv593882282=66, srv1293507742=8, srv1747400141=28, srv2039290169=44, srv1668284265=26, srv1246496094=6, srv1519314644=17, srv2107638748=46, srv378055747=56, srv702361237=78, srv444245511=61, srv637914902=71, srv277492397=50, srv306008607=51, srv521439280=65, srv809846139=87, srv1306074788=9, srv358881020=54, srv346695473=53, srv1367984024=13, srv344100082=52, srv1644793900=23, srv1843171822=35, srv1625445930=20, srv176538357=31, srv1805335123=33, srv872230395=89, srv1311668677=10, srv602092061=67, srv1693985027=27, srv604626877=68, srv1078276412=3, srv719589491=79, srv78414130=84, srv913586963=92, srv997073386=98, srv1067358804=2, srv942291463=95, srv2005405000=43} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-19 09:53:26,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-19 09:53:26,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-19 09:53:26,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-19 09:53:26,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-03-19 09:53:26,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-03-19 09:53:26,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-03-19 09:53:26,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-03-19 09:53:26,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-03-19 09:53:26,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-03-19 09:53:26,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-03-19 09:53:26,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-03-19 09:53:26,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-03-19 09:53:26,806 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0050929751922394705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007741618689438271); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0050929751922394705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:53:26,807 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-03-19 09:53:26,807 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.32115419086765E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0050929751922394705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007741618689438271); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0050929751922394705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-03-19 09:53:27,160 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 447 ms to try 5000 different iterations. Found a solution that moves 218 regions; Going from a computed imbalance of 8.32115419086765E-4 to a new imbalance of 3.5271795951300944E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.02422222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:53:27,161 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1940175115:83 , srv2005405000:83 , srv378055747:83 , srv483452625:83 , srv949993510:83 , srv997073386:83 , srv1130343184:84 , srv1242410403:84 , srv1306074788:84 , srv1312635970:84 , srv1367984024:84 , srv1440941716:84 , srv1513235373:84 , srv1519314644:84 , srv1550559515:84 , srv1625445930:84 , srv1644793900:84 , srv1693985027:84 , srv176317320:84 , srv176538357:84 , srv1774238949:84 , srv1906964914:84 , srv1917867363:84 , srv1958810416:84 , srv2001543385:84 , srv2039290169:84 , srv2107638748:84 , srv250877291:84 , srv273900190:84 , srv306008607:84 , srv344100082:84 , srv346695473:84 , srv358881020:84 , srv400302884:84 , srv418320518:84 , srv521439280:84 , srv605149346:84 , srv638367536:84 , srv648718789:84 , srv700159450:84 , srv702361237:84 , srv731118474:84 , srv736281688:84 , srv771698455:84 , srv78414130:84 , srv835880913:84 , srv872230395:84 , srv894681765:84 , srv940590657:84 , srv942291463:84 , srv966137735:84 , srv1034641065:85 , srv1059631235:85 , srv1067358804:85 , srv1078276412:85 , srv1246496094:85 , srv1281673890:85 , srv1293507742:85 , srv1311668677:85 , srv1512038533:85 , srv1549295911:85 , srv163817489:85 , srv1638298839:85 , srv1649399700:85 , srv1659054976:85 , srv1668284265:85 , srv1747400141:85 , srv1754514288:85 , srv1805335123:85 , srv1816073632:85 , srv1843171822:85 , srv19953607:85 , srv2099046634:85 , srv268346097:85 , srv374901804:85 , srv428984924:85 , srv429225318:85 , srv444245511:85 , srv472507203:85 , srv490713531:85 , srv593882282:85 , srv602092061:85 , srv604626877:85 , srv622923211:85 , srv637914902:85 , srv658317019:85 , srv682776345:85 , srv689068841:85 , srv719589491:85 , srv764526262:85 , srv791774291:85 , srv802360655:85 , srv911611954:85 , srv913586963:85 , srv921682066:85 , srv997665026:85 , srv1337035360:86 , srv1963283101:86 , srv809846139:86 , srv277492397:639 } 2024-03-19 09:53:27,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-19 09:53:27,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1649399700=24, srv700159450=77, srv1659054976=25, srv1774238949=32, srv1963283101=40, srv2001543385=42, srv648718789=73, srv802360655=86, srv997665026=99, srv966137735=97, srv658317019=74, srv472507203=62, srv268346097=48, srv1906964914=36, srv1512038533=15, srv1130343184=4, srv483452625=63, srv940590657=94, srv428984924=59, srv638367536=72, srv1034641065=0, srv1337035360=12, srv400302884=57, srv764526262=82, srv949993510=96, srv1917867363=37, srv1312635970=11, srv605149346=69, srv1059631235=1, srv622923211=70, srv1440941716=14, srv1550559515=19, srv791774291=85, srv771698455=83, srv1242410403=5, srv682776345=75, srv1816073632=34, srv490713531=64, srv835880913=88, srv1958810416=39, srv429225318=60, srv1281673890=7, srv2099046634=45, srv176317320=30, srv736281688=81, srv1513235373=16, srv418320518=58, srv19953607=41, srv921682066=93, srv273900190=49, srv731118474=80, srv250877291=47, srv374901804=55, srv163817489=21, srv1549295911=18, srv1638298839=22, srv689068841=76, srv911611954=91, srv1940175115=38, srv894681765=90, srv1754514288=29, srv593882282=66, srv1293507742=8, srv1747400141=28, srv2039290169=44, srv1668284265=26, srv1246496094=6, srv1519314644=17, srv2107638748=46, srv378055747=56, srv702361237=78, srv444245511=61, srv637914902=71, srv277492397=50, srv306008607=51, srv521439280=65, srv809846139=87, srv1306074788=9, srv358881020=54, srv346695473=53, srv1367984024=13, srv344100082=52, srv1644793900=23, srv1843171822=35, srv1625445930=20, srv176538357=31, srv1805335123=33, srv872230395=89, srv1311668677=10, srv602092061=67, srv1693985027=27, srv604626877=68, srv1078276412=3, srv719589491=79, srv78414130=84, srv913586963=92, srv997073386=98, srv1067358804=2, srv942291463=95, srv2005405000=43} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-19 09:53:27,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-19 09:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-03-19 09:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-03-19 09:53:27,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-03-19 09:53:27,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-03-19 09:53:27,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-03-19 09:53:27,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-03-19 09:53:27,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-03-19 09:53:27,284 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:53:27,285 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-03-19 09:53:27,285 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.374532732543281E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-03-19 09:53:27,527 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 336 ms to try 5000 different iterations. Found a solution that moves 27 regions; Going from a computed imbalance of 3.374532732543281E-5 to a new imbalance of 6.587043713088231E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.003); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:53:27,527 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1337035360:83 , srv1513235373:83 , srv1963283101:83 , srv428984924:83 , srv809846139:83 , srv1034641065:84 , srv1059631235:84 , srv1067358804:84 , srv1078276412:84 , srv1130343184:84 , srv1242410403:84 , srv1246496094:84 , srv1281673890:84 , srv1293507742:84 , srv1306074788:84 , srv1311668677:84 , srv1312635970:84 , srv1367984024:84 , srv1440941716:84 , srv1512038533:84 , srv1519314644:84 , srv1549295911:84 , srv1550559515:84 , srv1625445930:84 , srv163817489:84 , srv1638298839:84 , srv1644793900:84 , srv1649399700:84 , srv1659054976:84 , srv1668284265:84 , srv1693985027:84 , srv1747400141:84 , srv1754514288:84 , srv176317320:84 , srv176538357:84 , srv1774238949:84 , srv1805335123:84 , srv1816073632:84 , srv1843171822:84 , srv1906964914:84 , srv1917867363:84 , srv1958810416:84 , srv19953607:84 , srv2001543385:84 , srv2005405000:84 , srv2039290169:84 , srv2099046634:84 , srv2107638748:84 , srv250877291:84 , srv268346097:84 , srv273900190:84 , srv344100082:84 , srv346695473:84 , srv358881020:84 , srv374901804:84 , srv378055747:84 , srv400302884:84 , srv429225318:84 , srv444245511:84 , srv472507203:84 , srv483452625:84 , srv490713531:84 , srv521439280:84 , srv593882282:84 , srv602092061:84 , srv604626877:84 , srv605149346:84 , srv622923211:84 , srv637914902:84 , srv638367536:84 , srv648718789:84 , srv658317019:84 , srv682776345:84 , srv689068841:84 , srv700159450:84 , srv702361237:84 , srv719589491:84 , srv731118474:84 , srv736281688:84 , srv764526262:84 , srv771698455:84 , srv78414130:84 , srv791774291:84 , srv802360655:84 , srv835880913:84 , srv872230395:84 , srv911611954:84 , srv913586963:84 , srv921682066:84 , srv940590657:84 , srv942291463:84 , srv949993510:84 , srv966137735:84 , srv997665026:84 , srv1940175115:85 , srv306008607:85 , srv418320518:85 , srv894681765:85 , srv997073386:85 , srv277492397:684 } 2024-03-19 09:53:27,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-19 09:53:27,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1649399700=24, srv700159450=77, srv1659054976=25, srv1774238949=32, srv1963283101=40, srv2001543385=42, srv648718789=73, srv802360655=86, srv997665026=99, srv966137735=97, srv658317019=74, srv472507203=62, srv268346097=48, srv1906964914=36, srv1512038533=15, srv1130343184=4, srv483452625=63, srv940590657=94, srv428984924=59, srv638367536=72, srv1034641065=0, srv1337035360=12, srv400302884=57, srv764526262=82, srv949993510=96, srv1917867363=37, srv1312635970=11, srv605149346=69, srv1059631235=1, srv622923211=70, srv1440941716=14, srv1550559515=19, srv791774291=85, srv771698455=83, srv1242410403=5, srv682776345=75, srv1816073632=34, srv490713531=64, srv835880913=88, srv1958810416=39, srv429225318=60, srv1281673890=7, srv2099046634=45, srv176317320=30, srv736281688=81, srv1513235373=16, srv418320518=58, srv19953607=41, srv921682066=93, srv273900190=49, srv731118474=80, srv250877291=47, srv374901804=55, srv163817489=21, srv1549295911=18, srv1638298839=22, srv689068841=76, srv911611954=91, srv1940175115=38, srv894681765=90, srv1754514288=29, srv593882282=66, srv1293507742=8, srv1747400141=28, srv2039290169=44, srv1668284265=26, srv1246496094=6, srv1519314644=17, srv2107638748=46, srv378055747=56, srv702361237=78, srv444245511=61, srv637914902=71, srv277492397=50, srv306008607=51, srv521439280=65, srv809846139=87, srv1306074788=9, srv358881020=54, srv346695473=53, srv1367984024=13, srv344100082=52, srv1644793900=23, srv1843171822=35, srv1625445930=20, srv176538357=31, srv1805335123=33, srv872230395=89, srv1311668677=10, srv602092061=67, srv1693985027=27, srv604626877=68, srv1078276412=3, srv719589491=79, srv78414130=84, srv913586963=92, srv997073386=98, srv1067358804=2, srv942291463=95, srv2005405000=43} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-19 09:53:27,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-19 09:53:27,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-19 09:53:27,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-19 09:53:27,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-19 09:53:27,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-19 09:53:27,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-19 09:53:27,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-03-19 09:53:27,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-03-19 09:53:27,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-03-19 09:53:27,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-03-19 09:53:27,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-03-19 09:53:27,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-03-19 09:53:27,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-03-19 09:53:27,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-03-19 09:53:27,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-03-19 09:53:27,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-03-19 09:53:27,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-03-19 09:53:27,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-03-19 09:53:27,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-03-19 09:53:27,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-03-19 09:53:27,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-03-19 09:53:27,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-03-19 09:53:27,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-03-19 09:53:27,667 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:53:27,667 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1337035360:83 , srv1513235373:83 , srv1963283101:83 , srv428984924:83 , srv809846139:83 , srv1034641065:84 , srv1059631235:84 , srv1067358804:84 , srv1078276412:84 , srv1130343184:84 , srv1242410403:84 , srv1246496094:84 , srv1281673890:84 , srv1293507742:84 , srv1306074788:84 , srv1311668677:84 , srv1312635970:84 , srv1367984024:84 , srv1440941716:84 , srv1512038533:84 , srv1519314644:84 , srv1549295911:84 , srv1550559515:84 , srv1625445930:84 , srv163817489:84 , srv1638298839:84 , srv1644793900:84 , srv1649399700:84 , srv1659054976:84 , srv1668284265:84 , srv1693985027:84 , srv1747400141:84 , srv1754514288:84 , srv176317320:84 , srv176538357:84 , srv1774238949:84 , srv1805335123:84 , srv1816073632:84 , srv1843171822:84 , srv1906964914:84 , srv1917867363:84 , srv1958810416:84 , srv19953607:84 , srv2001543385:84 , srv2005405000:84 , srv2039290169:84 , srv2099046634:84 , srv2107638748:84 , srv250877291:84 , srv268346097:84 , srv273900190:84 , srv344100082:84 , srv346695473:84 , srv358881020:84 , srv374901804:84 , srv378055747:84 , srv400302884:84 , srv429225318:84 , srv444245511:84 , srv472507203:84 , srv483452625:84 , srv490713531:84 , srv521439280:84 , srv593882282:84 , srv602092061:84 , srv604626877:84 , srv605149346:84 , srv622923211:84 , srv637914902:84 , srv638367536:84 , srv648718789:84 , srv658317019:84 , srv682776345:84 , srv689068841:84 , srv700159450:84 , srv702361237:84 , srv719589491:84 , srv731118474:84 , srv736281688:84 , srv764526262:84 , srv771698455:84 , srv78414130:84 , srv791774291:84 , srv802360655:84 , srv835880913:84 , srv872230395:84 , srv911611954:84 , srv913586963:84 , srv921682066:84 , srv940590657:84 , srv942291463:84 , srv949993510:84 , srv966137735:84 , srv997665026:84 , srv1940175115:85 , srv306008607:85 , srv418320518:85 , srv894681765:85 , srv997073386:85 , srv277492397:684 } 2024-03-19 09:53:27,696 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=260 (was 248) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=3540 (was 4044) 2024-03-19 09:53:27,706 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=260, ProcessCount=173, AvailableMemoryMB=3539 2024-03-19 09:53:27,707 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-03-19 09:53:27,707 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-03-19 09:53:27,717 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1044861782:84 , srv1095918641:84 , srv1173173626:84 , srv1307266160:84 , srv1420657218:84 , srv1454402584:84 , srv1522465853:84 , srv1627704957:84 , srv1633717237:84 , srv1657763772:84 , srv1818180719:84 , srv1856221977:84 , srv1883797541:84 , srv1954538649:84 , srv2031907282:84 , srv2121189576:84 , srv427889764:84 , srv474749843:84 , srv576001827:84 , srv628089911:84 , srv640727305:84 , srv679770021:84 , srv705868498:84 , srv719205646:84 , srv743640926:84 , srv77267534:84 , srv837073702:84 , srv865628068:84 , srv99238558:84 , srv1017424056:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-03-19 09:53:27,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-19 09:53:27,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2031907282=15, srv1173173626=3, srv1454402584=6, srv1522465853=7, srv1856221977=12, srv705868498=23, srv1627704957=8, srv743640926=25, srv1044861782=1, srv1633717237=9, srv576001827=19, srv640727305=21, srv77267534=26, srv837073702=27, srv99238558=29, srv1095918641=2, srv427889764=17, srv474749843=18, srv679770021=22, srv1818180719=11, srv628089911=20, srv1883797541=13, srv1954538649=14, srv719205646=24, srv865628068=28, srv2121189576=16, srv1657763772=10, srv1017424056=0, srv1420657218=5, srv1307266160=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-19 09:53:27,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-19 09:53:27,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-19 09:53:27,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-19 09:53:27,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-19 09:53:27,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-19 09:53:27,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-19 09:53:27,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-19 09:53:27,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-19 09:53:27,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-19 09:53:27,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-19 09:53:27,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-19 09:53:27,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-19 09:53:27,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-19 09:53:27,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-03-19 09:53:27,749 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1015842839050616, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:53:27,749 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912513132025833, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1015842839050616, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-03-19 09:55:27,737 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5455605 different iterations. Found a solution that moves 1864 regions; Going from a computed imbalance of 0.9912513132025833 to a new imbalance of 4.3506689887128686E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6903703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:55:27,744 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1017424056:90 , srv1044861782:90 , srv1095918641:90 , srv1173173626:90 , srv1307266160:90 , srv1420657218:90 , srv1454402584:90 , srv1522465853:90 , srv1627704957:90 , srv1633717237:90 , srv1657763772:90 , srv1818180719:90 , srv1856221977:90 , srv1883797541:90 , srv1954538649:90 , srv2031907282:90 , srv2121189576:90 , srv427889764:90 , srv474749843:90 , srv576001827:90 , srv628089911:90 , srv640727305:90 , srv679770021:90 , srv705868498:90 , srv719205646:90 , srv743640926:90 , srv77267534:90 , srv837073702:90 , srv865628068:90 , srv99238558:90 } 2024-03-19 09:55:27,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-03-19 09:55:27,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2031907282=15, srv1173173626=3, srv1454402584=6, srv1522465853=7, srv1856221977=12, srv705868498=23, srv1627704957=8, srv743640926=25, srv1044861782=1, srv1633717237=9, srv576001827=19, srv640727305=21, srv77267534=26, srv837073702=27, srv99238558=29, srv1095918641=2, srv427889764=17, srv474749843=18, srv679770021=22, srv1818180719=11, srv628089911=20, srv1883797541=13, srv1954538649=14, srv719205646=24, srv865628068=28, srv2121189576=16, srv1657763772=10, srv1017424056=0, srv1420657218=5, srv1307266160=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-03-19 09:55:27,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-03-19 09:55:27,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-03-19 09:55:27,767 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-03-19 09:55:27,768 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1017424056:90 , srv1044861782:90 , srv1095918641:90 , srv1173173626:90 , srv1307266160:90 , srv1420657218:90 , srv1454402584:90 , srv1522465853:90 , srv1627704957:90 , srv1633717237:90 , srv1657763772:90 , srv1818180719:90 , srv1856221977:90 , srv1883797541:90 , srv1954538649:90 , srv2031907282:90 , srv2121189576:90 , srv427889764:90 , srv474749843:90 , srv576001827:90 , srv628089911:90 , srv640727305:90 , srv679770021:90 , srv705868498:90 , srv719205646:90 , srv743640926:90 , srv77267534:90 , srv837073702:90 , srv865628068:90 , srv99238558:90 } 2024-03-19 09:55:27,785 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=166 (was 260), ProcessCount=169 (was 173), AvailableMemoryMB=2788 (was 3539)