2024-02-27 08:53:16,877 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-02-27 08:53:17,217 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-02-27 08:53:17,336 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-02-27 08:53:17,404 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-27 08:53:17,413 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-27 08:53:17,430 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=273, ProcessCount=171, AvailableMemoryMB=11648 2024-02-27 08:53:17,433 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-27 08:53:17,433 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-27 08:53:17,573 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004934161:84 , srv1005521261:84 , srv1009993954:84 , srv1011044421:84 , srv104886962:84 , srv1093563684:84 , srv1106911535:84 , srv1123476490:84 , srv1126095508:84 , srv1127586324:84 , srv1134971447:84 , srv1138895780:84 , srv1177193734:84 , srv1185243579:84 , srv1188883415:84 , srv1223751615:84 , srv1230998339:84 , srv1234760483:84 , srv1274499508:84 , srv1278263717:84 , srv1280453896:84 , srv1280901594:84 , srv1324331199:84 , srv132600137:84 , srv1326332629:84 , srv1336464406:84 , srv1340501809:84 , srv135710056:84 , srv1358550087:84 , srv1360868508:84 , srv136785100:84 , srv1385515636:84 , srv1400839039:84 , srv1459239446:84 , srv1462820697:84 , srv1468052577:84 , srv1487164761:84 , srv1510987309:84 , srv1598826643:84 , srv1617097989:84 , srv161871734:84 , srv1620214187:84 , srv168384039:84 , srv169981611:84 , srv1719876384:84 , srv172712820:84 , srv1729545798:84 , srv1746427013:84 , srv1798865594:84 , srv1801733303:84 , srv1822754193:84 , srv1885381025:84 , srv1899991722:84 , srv1919030314:84 , srv1942435425:84 , srv1961804019:84 , srv1970757886:84 , srv1975964960:84 , srv200553115:84 , srv2024521212:84 , srv2050844541:84 , srv2057841823:84 , srv2071569682:84 , srv2072211086:84 , srv20859631:84 , srv2110874548:84 , srv2112831511:84 , srv238478196:84 , srv247750566:84 , srv275241283:84 , srv298285889:84 , srv298993442:84 , srv323993041:84 , srv353677105:84 , srv378922592:84 , srv384026313:84 , srv387309196:84 , srv527249129:84 , srv528572293:84 , srv535853360:84 , srv57493588:84 , srv609822360:84 , srv623732575:84 , srv665003425:84 , srv687155811:84 , srv690467333:84 , srv705763833:84 , srv753759648:84 , srv771511801:84 , srv797101667:84 , srv820465273:84 , srv833009251:84 , srv842533656:84 , srv885206895:84 , srv886662884:84 , srv903178131:84 , srv941070414:84 , srv988790085:84 , srv997735069:84 , srv1087281225:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-02-27 08:53:17,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-27 08:53:17,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1004934161=0, srv1087281225=5, srv1326332629=25, srv1009993954=2, srv1324331199=23, srv172712820=46, srv57493588=81, srv1360868508=30, srv1729545798=47, srv1123476490=8, srv200553115=59, srv1234760483=18, srv1138895780=12, srv1106911535=7, srv1280901594=22, srv2057841823=62, srv1177193734=13, srv941070414=97, srv527249129=78, srv1005521261=1, srv1127586324=10, srv797101667=90, srv1185243579=14, srv247750566=69, srv997735069=99, srv2072211086=64, srv323993041=73, srv833009251=92, srv1620214187=42, srv136785100=31, srv353677105=74, srv2071569682=63, srv1223751615=16, srv1340501809=27, srv705763833=87, srv1719876384=45, srv687155811=85, srv384026313=76, srv132600137=24, srv1975964960=58, srv535853360=80, srv2050844541=61, srv623732575=83, srv886662884=95, srv2110874548=66, srv298993442=72, srv1280453896=21, srv238478196=68, srv988790085=98, srv1134971447=11, srv528572293=79, srv1274499508=19, srv387309196=77, srv1011044421=3, srv1093563684=6, srv1617097989=40, srv1188883415=15, srv1942435425=55, srv1462820697=35, srv1487164761=37, srv842533656=93, srv378922592=75, srv885206895=94, srv168384039=43, srv169981611=44, srv104886962=4, srv1746427013=48, srv1468052577=36, srv1400839039=33, srv771511801=89, srv1919030314=54, srv1278263717=20, srv1885381025=52, srv1801733303=50, srv1798865594=49, srv903178131=96, srv135710056=28, srv275241283=70, srv665003425=84, srv690467333=86, srv2024521212=60, srv1336464406=26, srv1899991722=53, srv609822360=82, srv1385515636=32, srv161871734=41, srv2112831511=67, srv1358550087=29, srv20859631=65, srv1230998339=17, srv1459239446=34, srv1822754193=51, srv1510987309=38, srv1970757886=57, srv820465273=91, srv753759648=88, srv1126095508=9, srv1598826643=39, srv298285889=71, srv1961804019=56} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-27 08:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-27 08:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-27 08:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-27 08:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-27 08:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-27 08:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-27 08:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-27 08:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-27 08:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-27 08:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-27 08:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-27 08:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-27 08:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-27 08:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-27 08:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-27 08:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-27 08:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-27 08:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-27 08:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-27 08:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-27 08:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-27 08:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-27 08:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-27 08:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-27 08:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-27 08:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-27 08:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-27 08:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-27 08:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-27 08:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-27 08:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-27 08:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-27 08:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-27 08:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-27 08:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-27 08:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-27 08:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-27 08:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-27 08:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-27 08:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-27 08:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-27 08:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-27 08:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-27 08:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-27 08:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-27 08:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-27 08:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-27 08:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-27 08:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-27 08:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-27 08:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-27 08:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-27 08:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-27 08:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-27 08:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-27 08:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-27 08:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-27 08:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-27 08:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-27 08:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-27 08:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-27 08:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-27 08:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-27 08:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-27 08:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-27 08:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-27 08:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-27 08:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-27 08:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-27 08:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-27 08:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-27 08:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-27 08:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-27 08:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-27 08:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-27 08:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-27 08:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-27 08:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-27 08:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-27 08:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-27 08:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-27 08:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-27 08:53:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-27 08:53:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-27 08:53:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-27 08:53:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-27 08:53:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-27 08:53:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-27 08:53:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-27 08:53:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-27 08:53:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-27 08:53:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-27 08:53:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-27 08:53:17,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-27 08:53:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-27 08:53:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-27 08:53:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-27 08:53:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-27 08:53:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-27 08:53:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-27 08:53:17,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-27 08:53:17,863 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:53:17,864 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-27 08:53:17,864 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-27 08:53:18,246 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 541 ms to try 5000 different iterations. Found a solution that moves 4665 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17978270638635738. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04080985872492421, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056857472385830686, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5183333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04080985872492421, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17516666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23991666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:53:18,269 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1009993954:81 , srv1011044421:81 , srv1126095508:81 , srv690467333:81 , srv104886962:82 , srv132600137:82 , srv1617097989:82 , srv1822754193:82 , srv1961804019:82 , srv2072211086:82 , srv1280453896:83 , srv1326332629:83 , srv1340501809:83 , srv1510987309:83 , srv1801733303:83 , srv2057841823:83 , srv833009251:83 , srv1123476490:84 , srv1230998339:84 , srv1274499508:84 , srv135710056:84 , srv1468052577:84 , srv1487164761:84 , srv1719876384:84 , srv1899991722:84 , srv2024521212:84 , srv665003425:84 , srv820465273:84 , srv1185243579:85 , srv1234760483:85 , srv1324331199:85 , srv1336464406:85 , srv136785100:85 , srv1385515636:85 , srv1459239446:85 , srv1462820697:85 , srv161871734:85 , srv169981611:85 , srv1942435425:85 , srv1970757886:85 , srv2050844541:85 , srv2112831511:85 , srv298993442:85 , srv528572293:85 , srv535853360:85 , srv609822360:85 , srv771511801:85 , srv886662884:85 , srv1127586324:86 , srv1134971447:86 , srv1138895780:86 , srv1278263717:86 , srv1280901594:86 , srv1360868508:86 , srv1885381025:86 , srv1919030314:86 , srv200553115:86 , srv2071569682:86 , srv298285889:86 , srv323993041:86 , srv687155811:86 , srv797101667:86 , srv842533656:86 , srv997735069:86 , srv1005521261:87 , srv1093563684:87 , srv384026313:87 , srv527249129:87 , srv903178131:87 , srv1106911535:88 , srv1223751615:88 , srv1358550087:88 , srv1746427013:88 , srv1975964960:88 , srv2110874548:88 , srv238478196:88 , srv705763833:88 , srv753759648:88 , srv885206895:88 , srv1798865594:89 , srv247750566:89 , srv275241283:89 , srv988790085:89 , srv1004934161:90 , srv1598826643:90 , srv1620214187:90 , srv387309196:90 , srv941070414:90 , srv20859631:91 , srv623732575:91 , srv1188883415:92 , srv1400839039:92 , srv172712820:92 , srv378922592:92 , srv1177193734:94 , srv353677105:94 , srv168384039:96 , srv57493588:97 , srv1729545798:99 , srv1087281225:452 } 2024-02-27 08:53:18,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-27 08:53:18,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1004934161=0, srv1087281225=5, srv1326332629=25, srv1009993954=2, srv1324331199=23, srv172712820=46, srv57493588=81, srv1360868508=30, srv1729545798=47, srv1123476490=8, srv200553115=59, srv1234760483=18, srv1138895780=12, srv1106911535=7, srv1280901594=22, srv2057841823=62, srv1177193734=13, srv941070414=97, srv527249129=78, srv1005521261=1, srv1127586324=10, srv797101667=90, srv1185243579=14, srv247750566=69, srv997735069=99, srv2072211086=64, srv323993041=73, srv833009251=92, srv1620214187=42, srv136785100=31, srv353677105=74, srv2071569682=63, srv1223751615=16, srv1340501809=27, srv705763833=87, srv1719876384=45, srv687155811=85, srv384026313=76, srv132600137=24, srv1975964960=58, srv535853360=80, srv2050844541=61, srv623732575=83, srv886662884=95, srv2110874548=66, srv298993442=72, srv1280453896=21, srv238478196=68, srv988790085=98, srv1134971447=11, srv528572293=79, srv1274499508=19, srv387309196=77, srv1011044421=3, srv1093563684=6, srv1617097989=40, srv1188883415=15, srv1942435425=55, srv1462820697=35, srv1487164761=37, srv842533656=93, srv378922592=75, srv885206895=94, srv168384039=43, srv169981611=44, srv104886962=4, srv1746427013=48, srv1468052577=36, srv1400839039=33, srv771511801=89, srv1919030314=54, srv1278263717=20, srv1885381025=52, srv1801733303=50, srv1798865594=49, srv903178131=96, srv135710056=28, srv275241283=70, srv665003425=84, srv690467333=86, srv2024521212=60, srv1336464406=26, srv1899991722=53, srv609822360=82, srv1385515636=32, srv161871734=41, srv2112831511=67, srv1358550087=29, srv20859631=65, srv1230998339=17, srv1459239446=34, srv1822754193=51, srv1510987309=38, srv1970757886=57, srv820465273=91, srv753759648=88, srv1126095508=9, srv1598826643=39, srv298285889=71, srv1961804019=56} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-27 08:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-27 08:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-27 08:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-27 08:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-27 08:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-27 08:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-27 08:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-27 08:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-27 08:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-27 08:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-27 08:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-27 08:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-27 08:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-27 08:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-27 08:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-27 08:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-27 08:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-27 08:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-27 08:53:18,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-27 08:53:18,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-27 08:53:18,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-27 08:53:18,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-27 08:53:18,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-27 08:53:18,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-27 08:53:18,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-27 08:53:18,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-27 08:53:18,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-27 08:53:18,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-27 08:53:18,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-27 08:53:18,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-27 08:53:18,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-27 08:53:18,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-27 08:53:18,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-27 08:53:18,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-27 08:53:18,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-27 08:53:18,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-27 08:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-27 08:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-27 08:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-27 08:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-27 08:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-27 08:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-27 08:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-27 08:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-27 08:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-27 08:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-27 08:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-27 08:53:18,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-27 08:53:18,474 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04080985872492421, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056857472385830686, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04080985872492421, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17516666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23991666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:53:18,474 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-27 08:53:18,474 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17975004135819372, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04080985872492421, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056857472385830686, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04080985872492421, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17516666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23991666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-27 08:53:18,732 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 410 ms to try 5000 different iterations. Found a solution that moves 2636 regions; Going from a computed imbalance of 0.17975004135819372 to a new imbalance of 0.016488569640998037. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02272934271655747); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03791419220845018, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29288888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02272934271655747); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01325); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04733333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:53:18,746 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1729545798:74 , srv57493588:74 , srv1177193734:76 , srv168384039:76 , srv353677105:77 , srv1004934161:80 , srv378922592:80 , srv1620214187:81 , srv172712820:81 , srv1400839039:82 , srv1598826643:82 , srv1746427013:82 , srv387309196:82 , srv753759648:82 , srv941070414:82 , srv988790085:82 , srv1188883415:83 , srv1358550087:83 , srv275241283:83 , srv623732575:83 , srv885206895:83 , srv1280901594:84 , srv20859631:84 , srv2110874548:84 , srv2112831511:84 , srv238478196:84 , srv247750566:84 , srv298285889:84 , srv687155811:84 , srv705763833:84 , srv1106911535:85 , srv1127586324:85 , srv1970757886:85 , srv200553115:85 , srv384026313:85 , srv609822360:85 , srv665003425:85 , srv903178131:85 , srv997735069:85 , srv1005521261:86 , srv1093563684:86 , srv1123476490:86 , srv135710056:86 , srv1360868508:86 , srv136785100:86 , srv1885381025:86 , srv1919030314:86 , srv1975964960:86 , srv298993442:86 , srv323993041:86 , srv535853360:86 , srv820465273:86 , srv1134971447:87 , srv1138895780:87 , srv1223751615:87 , srv1326332629:87 , srv1336464406:87 , srv1459239446:87 , srv1462820697:87 , srv169981611:87 , srv1719876384:87 , srv1798865594:87 , srv1899991722:87 , srv2024521212:87 , srv2050844541:87 , srv2071569682:87 , srv527249129:87 , srv528572293:87 , srv797101667:87 , srv842533656:87 , srv104886962:88 , srv1185243579:88 , srv1234760483:88 , srv1274499508:88 , srv1324331199:88 , srv1510987309:88 , srv1942435425:88 , srv2057841823:88 , srv2072211086:88 , srv833009251:88 , srv1278263717:89 , srv1280453896:89 , srv132600137:89 , srv1340501809:89 , srv1385515636:89 , srv1487164761:89 , srv161871734:89 , srv1801733303:89 , srv1961804019:89 , srv771511801:89 , srv886662884:89 , srv1011044421:90 , srv1468052577:90 , srv1617097989:90 , srv1822754193:90 , srv1009993954:91 , srv1126095508:91 , srv1230998339:91 , srv690467333:91 , srv1087281225:524 } 2024-02-27 08:53:18,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-27 08:53:18,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1004934161=0, srv1087281225=5, srv1326332629=25, srv1009993954=2, srv1324331199=23, srv172712820=46, srv57493588=81, srv1360868508=30, srv1729545798=47, srv1123476490=8, srv200553115=59, srv1234760483=18, srv1138895780=12, srv1106911535=7, srv1280901594=22, srv2057841823=62, srv1177193734=13, srv941070414=97, srv527249129=78, srv1005521261=1, srv1127586324=10, srv797101667=90, srv1185243579=14, srv247750566=69, srv997735069=99, srv2072211086=64, srv323993041=73, srv833009251=92, srv1620214187=42, srv136785100=31, srv353677105=74, srv2071569682=63, srv1223751615=16, srv1340501809=27, srv705763833=87, srv1719876384=45, srv687155811=85, srv384026313=76, srv132600137=24, srv1975964960=58, srv535853360=80, srv2050844541=61, srv623732575=83, srv886662884=95, srv2110874548=66, srv298993442=72, srv1280453896=21, srv238478196=68, srv988790085=98, srv1134971447=11, srv528572293=79, srv1274499508=19, srv387309196=77, srv1011044421=3, srv1093563684=6, srv1617097989=40, srv1188883415=15, srv1942435425=55, srv1462820697=35, srv1487164761=37, srv842533656=93, srv378922592=75, srv885206895=94, srv168384039=43, srv169981611=44, srv104886962=4, srv1746427013=48, srv1468052577=36, srv1400839039=33, srv771511801=89, srv1919030314=54, srv1278263717=20, srv1885381025=52, srv1801733303=50, srv1798865594=49, srv903178131=96, srv135710056=28, srv275241283=70, srv665003425=84, srv690467333=86, srv2024521212=60, srv1336464406=26, srv1899991722=53, srv609822360=82, srv1385515636=32, srv161871734=41, srv2112831511=67, srv1358550087=29, srv20859631=65, srv1230998339=17, srv1459239446=34, srv1822754193=51, srv1510987309=38, srv1970757886=57, srv820465273=91, srv753759648=88, srv1126095508=9, srv1598826643=39, srv298285889=71, srv1961804019=56} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-27 08:53:18,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-27 08:53:18,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-27 08:53:18,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-27 08:53:18,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-27 08:53:18,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-27 08:53:18,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-27 08:53:18,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-27 08:53:18,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-27 08:53:18,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-27 08:53:18,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-27 08:53:18,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-27 08:53:18,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-27 08:53:18,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-27 08:53:18,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-27 08:53:18,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-27 08:53:18,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-27 08:53:18,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-27 08:53:18,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-27 08:53:18,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-27 08:53:18,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-27 08:53:18,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-27 08:53:18,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-27 08:53:18,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-27 08:53:18,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-27 08:53:18,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-27 08:53:18,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-27 08:53:18,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-27 08:53:18,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-27 08:53:18,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-27 08:53:18,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-27 08:53:18,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-27 08:53:18,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-27 08:53:18,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-27 08:53:18,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-27 08:53:18,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-27 08:53:18,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-27 08:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-27 08:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-27 08:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-27 08:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-27 08:53:18,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-27 08:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-27 08:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-27 08:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-27 08:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-27 08:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-27 08:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-27 08:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-27 08:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-27 08:53:18,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-27 08:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-27 08:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-27 08:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-27 08:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-27 08:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-27 08:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-27 08:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-27 08:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-27 08:53:18,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-27 08:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-27 08:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-27 08:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-27 08:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-27 08:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-27 08:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-27 08:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-27 08:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-27 08:53:18,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-27 08:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-27 08:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-27 08:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-27 08:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-27 08:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-27 08:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-27 08:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-27 08:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-27 08:53:18,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-27 08:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-27 08:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-27 08:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-27 08:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-27 08:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-27 08:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-27 08:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-27 08:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-27 08:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-27 08:53:18,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-27 08:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-27 08:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-27 08:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-27 08:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-27 08:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-27 08:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-27 08:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-27 08:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-27 08:53:18,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-27 08:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-27 08:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-27 08:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-27 08:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-27 08:53:18,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-27 08:53:18,891 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02272934271655747); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03791419220845018, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02272934271655747); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01325); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04733333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:53:18,891 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-27 08:53:18,891 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01647011197449442, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02272934271655747); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03791419220845018, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02272934271655747); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01325); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04733333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-27 08:53:19,290 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 496 ms to try 5000 different iterations. Found a solution that moves 992 regions; Going from a computed imbalance of 0.01647011197449442 to a new imbalance of 8.068679628840898E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006438288014889025); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00915019947284304); MoveCostFunction : (multiplier=7.0, imbalance=0.11022222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006438288014889025); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:53:19,299 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1223751615:81 , srv1400839039:82 , srv1798865594:82 , srv20859631:82 , srv1230998339:83 , srv1278263717:83 , srv1385515636:83 , srv1468052577:83 , srv161871734:83 , srv771511801:83 , srv886662884:83 , srv1093563684:84 , srv1185243579:84 , srv1188883415:84 , srv1462820697:84 , srv172712820:84 , srv1729545798:84 , srv1975964960:84 , srv2071569682:84 , srv247750566:84 , srv527249129:84 , srv623732575:84 , srv1005521261:85 , srv1009993954:85 , srv1106911535:85 , srv1126095508:85 , srv1134971447:85 , srv1138895780:85 , srv1234760483:85 , srv1274499508:85 , srv1280453896:85 , srv1324331199:85 , srv1336464406:85 , srv1360868508:85 , srv1487164761:85 , srv1598826643:85 , srv1617097989:85 , srv1942435425:85 , srv2110874548:85 , srv238478196:85 , srv275241283:85 , srv323993041:85 , srv384026313:85 , srv387309196:85 , srv528572293:85 , srv690467333:85 , srv797101667:85 , srv842533656:85 , srv903178131:85 , srv941070414:85 , srv1127586324:86 , srv132600137:86 , srv1340501809:86 , srv136785100:86 , srv1459239446:86 , srv1620214187:86 , srv168384039:86 , srv169981611:86 , srv1719876384:86 , srv1801733303:86 , srv1885381025:86 , srv1899991722:86 , srv1919030314:86 , srv200553115:86 , srv2050844541:86 , srv298993442:86 , srv378922592:86 , srv535853360:86 , srv705763833:86 , srv833009251:86 , srv885206895:86 , srv988790085:86 , srv997735069:86 , srv1004934161:87 , srv1011044421:87 , srv1177193734:87 , srv1280901594:87 , srv135710056:87 , srv1358550087:87 , srv1510987309:87 , srv1746427013:87 , srv1822754193:87 , srv1961804019:87 , srv1970757886:87 , srv2024521212:87 , srv2057841823:87 , srv298285889:87 , srv353677105:87 , srv609822360:87 , srv687155811:87 , srv753759648:87 , srv820465273:87 , srv1123476490:88 , srv1326332629:88 , srv2072211086:88 , srv2112831511:88 , srv57493588:88 , srv665003425:88 , srv104886962:90 , srv1087281225:539 } 2024-02-27 08:53:19,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-27 08:53:19,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1004934161=0, srv1087281225=5, srv1326332629=25, srv1009993954=2, srv1324331199=23, srv172712820=46, srv57493588=81, srv1360868508=30, srv1729545798=47, srv1123476490=8, srv200553115=59, srv1234760483=18, srv1138895780=12, srv1106911535=7, srv1280901594=22, srv2057841823=62, srv1177193734=13, srv941070414=97, srv527249129=78, srv1005521261=1, srv1127586324=10, srv797101667=90, srv1185243579=14, srv247750566=69, srv997735069=99, srv2072211086=64, srv323993041=73, srv833009251=92, srv1620214187=42, srv136785100=31, srv353677105=74, srv2071569682=63, srv1223751615=16, srv1340501809=27, srv705763833=87, srv1719876384=45, srv687155811=85, srv384026313=76, srv132600137=24, srv1975964960=58, srv535853360=80, srv2050844541=61, srv623732575=83, srv886662884=95, srv2110874548=66, srv298993442=72, srv1280453896=21, srv238478196=68, srv988790085=98, srv1134971447=11, srv528572293=79, srv1274499508=19, srv387309196=77, srv1011044421=3, srv1093563684=6, srv1617097989=40, srv1188883415=15, srv1942435425=55, srv1462820697=35, srv1487164761=37, srv842533656=93, srv378922592=75, srv885206895=94, srv168384039=43, srv169981611=44, srv104886962=4, srv1746427013=48, srv1468052577=36, srv1400839039=33, srv771511801=89, srv1919030314=54, srv1278263717=20, srv1885381025=52, srv1801733303=50, srv1798865594=49, srv903178131=96, srv135710056=28, srv275241283=70, srv665003425=84, srv690467333=86, srv2024521212=60, srv1336464406=26, srv1899991722=53, srv609822360=82, srv1385515636=32, srv161871734=41, srv2112831511=67, srv1358550087=29, srv20859631=65, srv1230998339=17, srv1459239446=34, srv1822754193=51, srv1510987309=38, srv1970757886=57, srv820465273=91, srv753759648=88, srv1126095508=9, srv1598826643=39, srv298285889=71, srv1961804019=56} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-27 08:53:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-27 08:53:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-27 08:53:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-27 08:53:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-27 08:53:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-27 08:53:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-27 08:53:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-27 08:53:19,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-27 08:53:19,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-27 08:53:19,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-27 08:53:19,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-27 08:53:19,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-27 08:53:19,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-27 08:53:19,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-27 08:53:19,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-27 08:53:19,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-27 08:53:19,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-27 08:53:19,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-27 08:53:19,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-27 08:53:19,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-27 08:53:19,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-27 08:53:19,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-27 08:53:19,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-27 08:53:19,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-27 08:53:19,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-27 08:53:19,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-27 08:53:19,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-27 08:53:19,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-27 08:53:19,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-27 08:53:19,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-27 08:53:19,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-27 08:53:19,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-27 08:53:19,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-27 08:53:19,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-27 08:53:19,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-27 08:53:19,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-27 08:53:19,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-27 08:53:19,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-27 08:53:19,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-27 08:53:19,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-27 08:53:19,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-27 08:53:19,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-27 08:53:19,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-27 08:53:19,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-27 08:53:19,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-27 08:53:19,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-27 08:53:19,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-27 08:53:19,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-27 08:53:19,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-27 08:53:19,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-27 08:53:19,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-27 08:53:19,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-27 08:53:19,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-27 08:53:19,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-27 08:53:19,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-27 08:53:19,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-27 08:53:19,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-27 08:53:19,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-27 08:53:19,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-27 08:53:19,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-27 08:53:19,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-27 08:53:19,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-27 08:53:19,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-27 08:53:19,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-27 08:53:19,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-27 08:53:19,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-27 08:53:19,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-27 08:53:19,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-27 08:53:19,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-27 08:53:19,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-27 08:53:19,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-27 08:53:19,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-27 08:53:19,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-27 08:53:19,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-27 08:53:19,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-27 08:53:19,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-27 08:53:19,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-27 08:53:19,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-27 08:53:19,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-27 08:53:19,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-27 08:53:19,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-27 08:53:19,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-27 08:53:19,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-27 08:53:19,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-27 08:53:19,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-27 08:53:19,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-27 08:53:19,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-27 08:53:19,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-27 08:53:19,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-27 08:53:19,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-27 08:53:19,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-27 08:53:19,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-27 08:53:19,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-27 08:53:19,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-27 08:53:19,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-27 08:53:19,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-27 08:53:19,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-27 08:53:19,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-27 08:53:19,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-27 08:53:19,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-27 08:53:19,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-27 08:53:19,493 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006438288014889025); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00915019947284304); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006438288014889025); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:53:19,493 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-27 08:53:19,493 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.999218304214238E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006438288014889025); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00915019947284304); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006438288014889025); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-27 08:53:19,863 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 507 ms to try 5000 different iterations. Found a solution that moves 309 regions; Going from a computed imbalance of 7.999218304214238E-4 to a new imbalance of 5.935595266277031E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.034333333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:53:19,864 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv104886962:82 , srv57493588:82 , srv1230998339:83 , srv1358550087:83 , srv1822754193:83 , srv1005521261:84 , srv1011044421:84 , srv1106911535:84 , srv1123476490:84 , srv1134971447:84 , srv1138895780:84 , srv1177193734:84 , srv1188883415:84 , srv1234760483:84 , srv1278263717:84 , srv1280453896:84 , srv1324331199:84 , srv1326332629:84 , srv1487164761:84 , srv1510987309:84 , srv168384039:84 , srv169981611:84 , srv1798865594:84 , srv1801733303:84 , srv1885381025:84 , srv1919030314:84 , srv1942435425:84 , srv1970757886:84 , srv1975964960:84 , srv200553115:84 , srv2024521212:84 , srv2050844541:84 , srv2057841823:84 , srv2072211086:84 , srv20859631:84 , srv353677105:84 , srv378922592:84 , srv527249129:84 , srv623732575:84 , srv705763833:84 , srv771511801:84 , srv797101667:84 , srv842533656:84 , srv885206895:84 , srv1004934161:85 , srv1009993954:85 , srv1093563684:85 , srv1126095508:85 , srv1127586324:85 , srv1185243579:85 , srv1274499508:85 , srv1280901594:85 , srv132600137:85 , srv1340501809:85 , srv135710056:85 , srv1360868508:85 , srv136785100:85 , srv1385515636:85 , srv1459239446:85 , srv1468052577:85 , srv1598826643:85 , srv161871734:85 , srv1620214187:85 , srv1719876384:85 , srv1729545798:85 , srv1746427013:85 , srv1899991722:85 , srv1961804019:85 , srv2071569682:85 , srv2112831511:85 , srv238478196:85 , srv247750566:85 , srv275241283:85 , srv298285889:85 , srv298993442:85 , srv323993041:85 , srv384026313:85 , srv387309196:85 , srv528572293:85 , srv535853360:85 , srv609822360:85 , srv665003425:85 , srv687155811:85 , srv820465273:85 , srv833009251:85 , srv886662884:85 , srv903178131:85 , srv941070414:85 , srv988790085:85 , srv997735069:85 , srv1223751615:86 , srv1336464406:86 , srv1400839039:86 , srv1462820697:86 , srv1617097989:86 , srv172712820:86 , srv2110874548:86 , srv690467333:86 , srv753759648:86 , srv1087281225:627 } 2024-02-27 08:53:19,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-27 08:53:19,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1004934161=0, srv1087281225=5, srv1326332629=25, srv1009993954=2, srv1324331199=23, srv172712820=46, srv57493588=81, srv1360868508=30, srv1729545798=47, srv1123476490=8, srv200553115=59, srv1234760483=18, srv1138895780=12, srv1106911535=7, srv1280901594=22, srv2057841823=62, srv1177193734=13, srv941070414=97, srv527249129=78, srv1005521261=1, srv1127586324=10, srv797101667=90, srv1185243579=14, srv247750566=69, srv997735069=99, srv2072211086=64, srv323993041=73, srv833009251=92, srv1620214187=42, srv136785100=31, srv353677105=74, srv2071569682=63, srv1223751615=16, srv1340501809=27, srv705763833=87, srv1719876384=45, srv687155811=85, srv384026313=76, srv132600137=24, srv1975964960=58, srv535853360=80, srv2050844541=61, srv623732575=83, srv886662884=95, srv2110874548=66, srv298993442=72, srv1280453896=21, srv238478196=68, srv988790085=98, srv1134971447=11, srv528572293=79, srv1274499508=19, srv387309196=77, srv1011044421=3, srv1093563684=6, srv1617097989=40, srv1188883415=15, srv1942435425=55, srv1462820697=35, srv1487164761=37, srv842533656=93, srv378922592=75, srv885206895=94, srv168384039=43, srv169981611=44, srv104886962=4, srv1746427013=48, srv1468052577=36, srv1400839039=33, srv771511801=89, srv1919030314=54, srv1278263717=20, srv1885381025=52, srv1801733303=50, srv1798865594=49, srv903178131=96, srv135710056=28, srv275241283=70, srv665003425=84, srv690467333=86, srv2024521212=60, srv1336464406=26, srv1899991722=53, srv609822360=82, srv1385515636=32, srv161871734=41, srv2112831511=67, srv1358550087=29, srv20859631=65, srv1230998339=17, srv1459239446=34, srv1822754193=51, srv1510987309=38, srv1970757886=57, srv820465273=91, srv753759648=88, srv1126095508=9, srv1598826643=39, srv298285889=71, srv1961804019=56} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-27 08:53:19,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-27 08:53:19,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-27 08:53:19,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-27 08:53:19,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-27 08:53:19,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-27 08:53:19,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-27 08:53:19,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-27 08:53:19,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-27 08:53:19,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-27 08:53:19,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-27 08:53:19,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-27 08:53:19,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-27 08:53:19,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-27 08:53:19,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-27 08:53:19,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-27 08:53:19,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-27 08:53:19,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-27 08:53:19,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-27 08:53:19,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-27 08:53:19,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-27 08:53:19,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-27 08:53:19,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-27 08:53:19,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-27 08:53:19,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-27 08:53:19,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-27 08:53:19,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-27 08:53:19,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-27 08:53:19,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-27 08:53:19,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-27 08:53:19,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-27 08:53:19,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-27 08:53:19,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-27 08:53:19,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-27 08:53:19,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-27 08:53:19,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-27 08:53:19,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-27 08:53:19,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-27 08:53:19,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-27 08:53:19,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-27 08:53:19,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-27 08:53:19,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-27 08:53:19,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-27 08:53:19,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-27 08:53:19,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-27 08:53:19,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-27 08:53:19,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-27 08:53:19,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-27 08:53:19,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-27 08:53:19,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-27 08:53:19,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-27 08:53:19,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-27 08:53:19,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-27 08:53:19,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-27 08:53:19,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-27 08:53:19,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-27 08:53:19,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-27 08:53:19,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-27 08:53:19,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-27 08:53:19,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-27 08:53:19,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-27 08:53:19,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-27 08:53:19,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-27 08:53:19,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-27 08:53:19,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-27 08:53:19,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-27 08:53:19,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-27 08:53:19,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-27 08:53:19,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-27 08:53:19,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-27 08:53:19,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-27 08:53:19,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-27 08:53:19,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-27 08:53:19,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-27 08:53:19,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-27 08:53:19,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-27 08:53:19,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-27 08:53:19,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-27 08:53:19,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-27 08:53:19,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-27 08:53:19,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-27 08:53:19,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-27 08:53:19,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-27 08:53:19,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-27 08:53:19,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-27 08:53:19,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-27 08:53:19,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-27 08:53:19,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-27 08:53:19,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-27 08:53:19,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-27 08:53:19,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-27 08:53:19,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-27 08:53:19,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-27 08:53:19,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-27 08:53:19,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-27 08:53:19,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-27 08:53:19,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-27 08:53:19,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-27 08:53:19,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-27 08:53:19,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-27 08:53:19,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-27 08:53:19,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-27 08:53:20,058 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:53:20,058 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-27 08:53:20,058 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.719228841784712E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-27 08:53:20,307 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 391 ms to try 5000 different iterations. Found a solution that moves 37 regions; Going from a computed imbalance of 5.719228841784712E-5 to a new imbalance of 7.948501337768768E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.004111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:53:20,307 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1188883415:83 , srv1336464406:83 , srv1340501809:83 , srv1459239446:83 , srv1617097989:83 , srv172712820:83 , srv1961804019:83 , srv2110874548:83 , srv690467333:83 , srv753759648:83 , srv1004934161:84 , srv1005521261:84 , srv1009993954:84 , srv1011044421:84 , srv104886962:84 , srv1093563684:84 , srv1106911535:84 , srv1123476490:84 , srv1126095508:84 , srv1127586324:84 , srv1134971447:84 , srv1138895780:84 , srv1185243579:84 , srv1223751615:84 , srv1234760483:84 , srv1274499508:84 , srv1278263717:84 , srv1280901594:84 , srv1324331199:84 , srv132600137:84 , srv1326332629:84 , srv135710056:84 , srv1360868508:84 , srv136785100:84 , srv1385515636:84 , srv1400839039:84 , srv1462820697:84 , srv1468052577:84 , srv1487164761:84 , srv1510987309:84 , srv1598826643:84 , srv161871734:84 , srv1620214187:84 , srv168384039:84 , srv169981611:84 , srv1719876384:84 , srv1729545798:84 , srv1746427013:84 , srv1798865594:84 , srv1801733303:84 , srv1822754193:84 , srv1885381025:84 , srv1899991722:84 , srv1919030314:84 , srv1942435425:84 , srv1975964960:84 , srv2024521212:84 , srv2050844541:84 , srv2057841823:84 , srv2071569682:84 , srv2072211086:84 , srv2112831511:84 , srv238478196:84 , srv247750566:84 , srv275241283:84 , srv298285889:84 , srv298993442:84 , srv323993041:84 , srv353677105:84 , srv378922592:84 , srv384026313:84 , srv387309196:84 , srv527249129:84 , srv528572293:84 , srv535853360:84 , srv609822360:84 , srv623732575:84 , srv665003425:84 , srv687155811:84 , srv705763833:84 , srv797101667:84 , srv820465273:84 , srv833009251:84 , srv842533656:84 , srv886662884:84 , srv903178131:84 , srv941070414:84 , srv988790085:84 , srv997735069:84 , srv1177193734:85 , srv1230998339:85 , srv1280453896:85 , srv1358550087:85 , srv1970757886:85 , srv200553115:85 , srv20859631:85 , srv57493588:85 , srv771511801:85 , srv885206895:85 , srv1087281225:684 } 2024-02-27 08:53:20,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-27 08:53:20,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1004934161=0, srv1087281225=5, srv1326332629=25, srv1009993954=2, srv1324331199=23, srv172712820=46, srv57493588=81, srv1360868508=30, srv1729545798=47, srv1123476490=8, srv200553115=59, srv1234760483=18, srv1138895780=12, srv1106911535=7, srv1280901594=22, srv2057841823=62, srv1177193734=13, srv941070414=97, srv527249129=78, srv1005521261=1, srv1127586324=10, srv797101667=90, srv1185243579=14, srv247750566=69, srv997735069=99, srv2072211086=64, srv323993041=73, srv833009251=92, srv1620214187=42, srv136785100=31, srv353677105=74, srv2071569682=63, srv1223751615=16, srv1340501809=27, srv705763833=87, srv1719876384=45, srv687155811=85, srv384026313=76, srv132600137=24, srv1975964960=58, srv535853360=80, srv2050844541=61, srv623732575=83, srv886662884=95, srv2110874548=66, srv298993442=72, srv1280453896=21, srv238478196=68, srv988790085=98, srv1134971447=11, srv528572293=79, srv1274499508=19, srv387309196=77, srv1011044421=3, srv1093563684=6, srv1617097989=40, srv1188883415=15, srv1942435425=55, srv1462820697=35, srv1487164761=37, srv842533656=93, srv378922592=75, srv885206895=94, srv168384039=43, srv169981611=44, srv104886962=4, srv1746427013=48, srv1468052577=36, srv1400839039=33, srv771511801=89, srv1919030314=54, srv1278263717=20, srv1885381025=52, srv1801733303=50, srv1798865594=49, srv903178131=96, srv135710056=28, srv275241283=70, srv665003425=84, srv690467333=86, srv2024521212=60, srv1336464406=26, srv1899991722=53, srv609822360=82, srv1385515636=32, srv161871734=41, srv2112831511=67, srv1358550087=29, srv20859631=65, srv1230998339=17, srv1459239446=34, srv1822754193=51, srv1510987309=38, srv1970757886=57, srv820465273=91, srv753759648=88, srv1126095508=9, srv1598826643=39, srv298285889=71, srv1961804019=56} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-27 08:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-27 08:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-27 08:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-27 08:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-27 08:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-27 08:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-27 08:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-27 08:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-27 08:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-27 08:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-27 08:53:20,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-27 08:53:20,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-27 08:53:20,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-27 08:53:20,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-27 08:53:20,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-27 08:53:20,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-27 08:53:20,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-27 08:53:20,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-27 08:53:20,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-27 08:53:20,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-27 08:53:20,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-27 08:53:20,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-27 08:53:20,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-27 08:53:20,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-27 08:53:20,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-27 08:53:20,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-27 08:53:20,434 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:53:20,435 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1188883415:83 , srv1336464406:83 , srv1340501809:83 , srv1459239446:83 , srv1617097989:83 , srv172712820:83 , srv1961804019:83 , srv2110874548:83 , srv690467333:83 , srv753759648:83 , srv1004934161:84 , srv1005521261:84 , srv1009993954:84 , srv1011044421:84 , srv104886962:84 , srv1093563684:84 , srv1106911535:84 , srv1123476490:84 , srv1126095508:84 , srv1127586324:84 , srv1134971447:84 , srv1138895780:84 , srv1185243579:84 , srv1223751615:84 , srv1234760483:84 , srv1274499508:84 , srv1278263717:84 , srv1280901594:84 , srv1324331199:84 , srv132600137:84 , srv1326332629:84 , srv135710056:84 , srv1360868508:84 , srv136785100:84 , srv1385515636:84 , srv1400839039:84 , srv1462820697:84 , srv1468052577:84 , srv1487164761:84 , srv1510987309:84 , srv1598826643:84 , srv161871734:84 , srv1620214187:84 , srv168384039:84 , srv169981611:84 , srv1719876384:84 , srv1729545798:84 , srv1746427013:84 , srv1798865594:84 , srv1801733303:84 , srv1822754193:84 , srv1885381025:84 , srv1899991722:84 , srv1919030314:84 , srv1942435425:84 , srv1975964960:84 , srv2024521212:84 , srv2050844541:84 , srv2057841823:84 , srv2071569682:84 , srv2072211086:84 , srv2112831511:84 , srv238478196:84 , srv247750566:84 , srv275241283:84 , srv298285889:84 , srv298993442:84 , srv323993041:84 , srv353677105:84 , srv378922592:84 , srv384026313:84 , srv387309196:84 , srv527249129:84 , srv528572293:84 , srv535853360:84 , srv609822360:84 , srv623732575:84 , srv665003425:84 , srv687155811:84 , srv705763833:84 , srv797101667:84 , srv820465273:84 , srv833009251:84 , srv842533656:84 , srv886662884:84 , srv903178131:84 , srv941070414:84 , srv988790085:84 , srv997735069:84 , srv1177193734:85 , srv1230998339:85 , srv1280453896:85 , srv1358550087:85 , srv1970757886:85 , srv200553115:85 , srv20859631:85 , srv57493588:85 , srv771511801:85 , srv885206895:85 , srv1087281225:684 } 2024-02-27 08:53:20,475 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=283 (was 273) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=10874 (was 11648) 2024-02-27 08:53:20,484 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=283, ProcessCount=171, AvailableMemoryMB=10874 2024-02-27 08:53:20,485 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-27 08:53:20,485 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-02-27 08:53:20,496 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1032034655:84 , srv1041433510:84 , srv1112176199:84 , srv1165681870:84 , srv126609409:84 , srv1300971684:84 , srv1339101335:84 , srv1399038465:84 , srv1422410525:84 , srv1458636763:84 , srv1541231203:84 , srv1656623374:84 , srv1694706952:84 , srv1760118374:84 , srv1797907817:84 , srv1941793282:84 , srv2071737835:84 , srv229201146:84 , srv24369195:84 , srv325388319:84 , srv564287706:84 , srv65484404:84 , srv735856827:84 , srv739429179:84 , srv752243786:84 , srv801505504:84 , srv804627118:84 , srv820801275:84 , srv841700580:84 , srv930483161:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-02-27 08:53:20,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-27 08:53:20,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1300971684=5, srv24369195=18, srv1941793282=15, srv735856827=22, srv801505504=25, srv1422410525=8, srv804627118=26, srv739429179=23, srv1458636763=9, srv1797907817=14, srv325388319=19, srv1165681870=3, srv1541231203=10, srv1112176199=2, srv1041433510=1, srv564287706=20, srv1339101335=6, srv229201146=17, srv2071737835=16, srv752243786=24, srv1694706952=12, srv841700580=28, srv65484404=21, srv1032034655=0, srv126609409=4, srv1760118374=13, srv930483161=29, srv1656623374=11, srv1399038465=7, srv820801275=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-27 08:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-27 08:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-27 08:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-27 08:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-27 08:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-27 08:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-27 08:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-27 08:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-27 08:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-27 08:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-27 08:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-27 08:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-27 08:53:20,528 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0978691716880244, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:53:20,528 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912501425828546, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0978691716880244, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-02-27 08:55:20,515 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5798483 different iterations. Found a solution that moves 1860 regions; Going from a computed imbalance of 0.9912501425828546 to a new imbalance of 4.341332789166274E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6888888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:55:20,522 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1032034655:90 , srv1041433510:90 , srv1112176199:90 , srv1165681870:90 , srv126609409:90 , srv1300971684:90 , srv1339101335:90 , srv1399038465:90 , srv1422410525:90 , srv1458636763:90 , srv1541231203:90 , srv1656623374:90 , srv1694706952:90 , srv1760118374:90 , srv1797907817:90 , srv1941793282:90 , srv2071737835:90 , srv229201146:90 , srv24369195:90 , srv325388319:90 , srv564287706:90 , srv65484404:90 , srv735856827:90 , srv739429179:90 , srv752243786:90 , srv801505504:90 , srv804627118:90 , srv820801275:90 , srv841700580:90 , srv930483161:90 } 2024-02-27 08:55:20,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-27 08:55:20,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1300971684=5, srv24369195=18, srv1941793282=15, srv735856827=22, srv801505504=25, srv1422410525=8, srv804627118=26, srv739429179=23, srv1458636763=9, srv1797907817=14, srv325388319=19, srv1165681870=3, srv1541231203=10, srv1112176199=2, srv1041433510=1, srv564287706=20, srv1339101335=6, srv229201146=17, srv2071737835=16, srv752243786=24, srv1694706952=12, srv841700580=28, srv65484404=21, srv1032034655=0, srv126609409=4, srv1760118374=13, srv930483161=29, srv1656623374=11, srv1399038465=7, srv820801275=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-27 08:55:20,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-27 08:55:20,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-27 08:55:20,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-27 08:55:20,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-27 08:55:20,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-27 08:55:20,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-27 08:55:20,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-27 08:55:20,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-27 08:55:20,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-27 08:55:20,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-27 08:55:20,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-27 08:55:20,542 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-27 08:55:20,542 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1032034655:90 , srv1041433510:90 , srv1112176199:90 , srv1165681870:90 , srv126609409:90 , srv1300971684:90 , srv1339101335:90 , srv1399038465:90 , srv1422410525:90 , srv1458636763:90 , srv1541231203:90 , srv1656623374:90 , srv1694706952:90 , srv1760118374:90 , srv1797907817:90 , srv1941793282:90 , srv2071737835:90 , srv229201146:90 , srv24369195:90 , srv325388319:90 , srv564287706:90 , srv65484404:90 , srv735856827:90 , srv739429179:90 , srv752243786:90 , srv801505504:90 , srv804627118:90 , srv820801275:90 , srv841700580:90 , srv930483161:90 } 2024-02-27 08:55:20,556 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=200 (was 283), ProcessCount=168 (was 171), AvailableMemoryMB=10456 (was 10874)