2024-02-11 18:53:08,047 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-02-11 18:53:08,358 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-02-11 18:53:08,456 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-02-11 18:53:08,519 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-11 18:53:08,527 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-11 18:53:08,539 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=281, ProcessCount=168, AvailableMemoryMB=3291 2024-02-11 18:53:08,541 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-11 18:53:08,541 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-11 18:53:08,657 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1006627613:84 , srv1027925338:84 , srv1030835280:84 , srv1036285527:84 , srv1042179257:84 , srv1064131452:84 , srv1099488219:84 , srv1120722448:84 , srv1125781138:84 , srv1127037739:84 , srv1135179152:84 , srv1165743802:84 , srv1184758474:84 , srv1190852800:84 , srv1191212164:84 , srv1226047727:84 , srv123551959:84 , srv1263413818:84 , srv1274420825:84 , srv1297321800:84 , srv1342719351:84 , srv1345370869:84 , srv1360598204:84 , srv1423676235:84 , srv1467085638:84 , srv1479418491:84 , srv1492995673:84 , srv1495242089:84 , srv150575840:84 , srv1516218974:84 , srv1521119703:84 , srv1522921333:84 , srv1552738461:84 , srv1569119757:84 , srv1580694600:84 , srv1587626392:84 , srv1601350001:84 , srv16620023:84 , srv1662233129:84 , srv1666719485:84 , srv1677414980:84 , srv1701171541:84 , srv1743740137:84 , srv175915686:84 , srv1764544001:84 , srv1781448719:84 , srv1803164870:84 , srv1833267415:84 , srv1845423433:84 , srv1855183840:84 , srv1857574904:84 , srv1886914667:84 , srv1907038284:84 , srv1963137198:84 , srv1963987996:84 , srv2000998320:84 , srv2004367761:84 , srv2006142720:84 , srv2025243316:84 , srv2038649829:84 , srv2040070986:84 , srv2040188673:84 , srv2057462117:84 , srv2069926856:84 , srv2083170357:84 , srv2145535204:84 , srv25186063:84 , srv275649812:84 , srv29173197:84 , srv297757912:84 , srv301788563:84 , srv312142622:84 , srv321789151:84 , srv347924654:84 , srv356035052:84 , srv395334574:84 , srv418500352:84 , srv449749423:84 , srv485793388:84 , srv562271760:84 , srv56451041:84 , srv608913086:84 , srv628407792:84 , srv69062308:84 , srv710191511:84 , srv726185941:84 , srv74120033:84 , srv776935265:84 , srv839592942:84 , srv84005535:84 , srv846127248:84 , srv873603612:84 , srv874483997:84 , srv886561683:84 , srv891905771:84 , srv89525353:84 , srv915739005:84 , srv924461691:84 , srv936459373:84 , srv193968837:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-02-11 18:53:08,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 18:53:08,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1030835280=2, srv1963987996=55, srv321789151=73, srv1342719351=20, srv2006142720=58, srv1521119703=30, srv1099488219=6, srv1803164870=46, srv1190852800=13, srv2069926856=64, srv1495242089=27, srv1027925338=1, srv710191511=85, srv924461691=98, srv1263413818=17, srv1601350001=36, srv1907038284=52, srv1855183840=49, srv915739005=97, srv69062308=84, srv1552738461=32, srv1587626392=35, srv2040070986=61, srv874483997=93, srv1833267415=47, srv193968837=53, srv2025243316=59, srv1677414980=40, srv1042179257=4, srv1701171541=41, srv301788563=71, srv1467085638=24, srv1479418491=25, srv1423676235=23, srv297757912=70, srv2004367761=57, srv2083170357=65, srv84005535=90, srv936459373=99, srv1781448719=45, srv175915686=43, srv56451041=81, srv846127248=91, srv312142622=72, srv891905771=95, srv1184758474=12, srv1857574904=50, srv1963137198=54, srv1127037739=9, srv395334574=76, srv886561683=94, srv562271760=80, srv1662233129=38, srv1125781138=8, srv1516218974=29, srv1297321800=19, srv1845423433=48, srv2000998320=56, srv74120033=87, srv150575840=28, srv1743740137=42, srv1036285527=3, srv2038649829=60, srv628407792=83, srv1522921333=31, srv2145535204=66, srv726185941=86, srv1191212164=14, srv2040188673=62, srv275649812=68, srv1274420825=18, srv1064131452=5, srv776935265=88, srv608913086=82, srv2057462117=63, srv1120722448=7, srv29173197=69, srv1006627613=0, srv1345370869=21, srv1165743802=11, srv356035052=75, srv1666719485=39, srv89525353=96, srv1135179152=10, srv1360598204=22, srv1492995673=26, srv123551959=16, srv347924654=74, srv449749423=78, srv873603612=92, srv1580694600=34, srv839592942=89, srv25186063=67, srv1569119757=33, srv418500352=77, srv1764544001=44, srv16620023=37, srv485793388=79, srv1226047727=15, srv1886914667=51} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 18:53:08,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 18:53:08,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 18:53:08,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 18:53:08,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 18:53:08,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 18:53:08,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 18:53:08,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 18:53:08,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 18:53:08,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 18:53:08,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 18:53:08,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 18:53:08,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 18:53:08,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 18:53:08,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 18:53:08,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 18:53:08,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 18:53:08,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 18:53:08,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 18:53:08,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 18:53:08,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 18:53:08,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 18:53:08,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 18:53:08,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 18:53:08,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 18:53:08,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 18:53:08,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 18:53:08,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 18:53:08,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 18:53:08,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 18:53:08,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 18:53:08,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 18:53:08,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 18:53:08,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 18:53:08,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 18:53:08,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 18:53:08,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 18:53:08,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 18:53:08,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 18:53:08,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 18:53:08,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 18:53:08,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 18:53:08,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 18:53:08,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 18:53:08,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 18:53:08,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 18:53:08,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 18:53:08,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 18:53:08,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 18:53:08,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 18:53:08,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 18:53:08,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 18:53:08,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 18:53:08,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 18:53:08,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 18:53:08,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 18:53:08,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 18:53:08,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 18:53:08,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 18:53:08,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 18:53:08,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 18:53:08,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 18:53:08,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 18:53:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 18:53:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 18:53:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 18:53:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 18:53:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 18:53:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 18:53:08,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 18:53:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 18:53:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 18:53:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 18:53:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 18:53:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 18:53:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 18:53:08,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 18:53:08,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 18:53:08,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 18:53:08,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 18:53:08,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 18:53:08,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 18:53:08,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 18:53:08,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 18:53:08,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 18:53:08,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 18:53:08,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 18:53:08,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 18:53:08,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 18:53:08,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 18:53:08,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 18:53:08,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 18:53:08,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 18:53:08,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 18:53:08,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 18:53:08,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 18:53:08,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 18:53:08,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 18:53:08,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 18:53:08,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 18:53:08,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 18:53:08,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 18:53:09,128 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 18:53:09,128 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 18:53:09,128 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 18:53:09,866 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1065 ms to try 5000 different iterations. Found a solution that moves 4729 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17595522060515778. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03825244804935104, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057536403349859384, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5254444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03825244804935104, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17141666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.235, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 18:53:09,892 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv321789151:82 , srv1042179257:83 , srv1423676235:83 , srv1569119757:83 , srv1662233129:83 , srv1963987996:83 , srv2083170357:83 , srv395334574:83 , srv891905771:83 , srv1226047727:84 , srv1521119703:84 , srv1886914667:84 , srv1963137198:84 , srv2145535204:84 , srv275649812:84 , srv301788563:84 , srv347924654:84 , srv69062308:84 , srv89525353:84 , srv1036285527:85 , srv1135179152:85 , srv1191212164:85 , srv1297321800:85 , srv1345370869:85 , srv1360598204:85 , srv1467085638:85 , srv1495242089:85 , srv150575840:85 , srv1587626392:85 , srv175915686:85 , srv1857574904:85 , srv2004367761:85 , srv297757912:85 , srv873603612:85 , srv1006627613:86 , srv1120722448:86 , srv1125781138:86 , srv1184758474:86 , srv123551959:86 , srv1274420825:86 , srv1516218974:86 , srv1522921333:86 , srv16620023:86 , srv1764544001:86 , srv1803164870:86 , srv2000998320:86 , srv2057462117:86 , srv2069926856:86 , srv418500352:86 , srv608913086:86 , srv1099488219:87 , srv1342719351:87 , srv1479418491:87 , srv1492995673:87 , srv1552738461:87 , srv1677414980:87 , srv1743740137:87 , srv1781448719:87 , srv1833267415:87 , srv2040070986:87 , srv562271760:87 , srv56451041:87 , srv628407792:87 , srv846127248:87 , srv936459373:87 , srv1030835280:88 , srv1064131452:88 , srv1190852800:88 , srv1601350001:88 , srv1701171541:88 , srv1845423433:88 , srv1855183840:88 , srv1907038284:88 , srv2006142720:88 , srv2025243316:88 , srv2040188673:88 , srv25186063:88 , srv312142622:88 , srv356035052:88 , srv726185941:88 , srv776935265:88 , srv839592942:88 , srv84005535:88 , srv915739005:88 , srv924461691:88 , srv1027925338:89 , srv1127037739:89 , srv1165743802:89 , srv1580694600:89 , srv74120033:89 , srv1263413818:90 , srv2038649829:90 , srv29173197:90 , srv449749423:90 , srv1666719485:91 , srv485793388:91 , srv710191511:91 , srv886561683:92 , srv874483997:98 , srv193968837:430 } 2024-02-11 18:53:09,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 18:53:09,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1030835280=2, srv1963987996=55, srv321789151=73, srv1342719351=20, srv2006142720=58, srv1521119703=30, srv1099488219=6, srv1803164870=46, srv1190852800=13, srv2069926856=64, srv1495242089=27, srv1027925338=1, srv710191511=85, srv924461691=98, srv1263413818=17, srv1601350001=36, srv1907038284=52, srv1855183840=49, srv915739005=97, srv69062308=84, srv1552738461=32, srv1587626392=35, srv2040070986=61, srv874483997=93, srv1833267415=47, srv193968837=53, srv2025243316=59, srv1677414980=40, srv1042179257=4, srv1701171541=41, srv301788563=71, srv1467085638=24, srv1479418491=25, srv1423676235=23, srv297757912=70, srv2004367761=57, srv2083170357=65, srv84005535=90, srv936459373=99, srv1781448719=45, srv175915686=43, srv56451041=81, srv846127248=91, srv312142622=72, srv891905771=95, srv1184758474=12, srv1857574904=50, srv1963137198=54, srv1127037739=9, srv395334574=76, srv886561683=94, srv562271760=80, srv1662233129=38, srv1125781138=8, srv1516218974=29, srv1297321800=19, srv1845423433=48, srv2000998320=56, srv74120033=87, srv150575840=28, srv1743740137=42, srv1036285527=3, srv2038649829=60, srv628407792=83, srv1522921333=31, srv2145535204=66, srv726185941=86, srv1191212164=14, srv2040188673=62, srv275649812=68, srv1274420825=18, srv1064131452=5, srv776935265=88, srv608913086=82, srv2057462117=63, srv1120722448=7, srv29173197=69, srv1006627613=0, srv1345370869=21, srv1165743802=11, srv356035052=75, srv1666719485=39, srv89525353=96, srv1135179152=10, srv1360598204=22, srv1492995673=26, srv123551959=16, srv347924654=74, srv449749423=78, srv873603612=92, srv1580694600=34, srv839592942=89, srv25186063=67, srv1569119757=33, srv418500352=77, srv1764544001=44, srv16620023=37, srv485793388=79, srv1226047727=15, srv1886914667=51} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 18:53:09,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 18:53:09,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 18:53:09,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 18:53:09,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 18:53:09,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 18:53:09,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 18:53:09,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 18:53:09,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 18:53:09,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 18:53:09,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 18:53:09,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 18:53:09,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 18:53:09,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 18:53:09,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 18:53:09,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 18:53:09,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 18:53:09,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 18:53:09,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 18:53:09,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 18:53:09,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 18:53:09,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 18:53:09,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 18:53:09,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 18:53:09,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 18:53:09,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 18:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 18:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 18:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 18:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 18:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 18:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 18:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 18:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 18:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 18:53:09,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 18:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 18:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 18:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 18:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 18:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 18:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 18:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 18:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 18:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 18:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 18:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 18:53:09,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 18:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 18:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 18:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 18:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 18:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 18:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 18:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 18:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 18:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 18:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 18:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 18:53:09,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 18:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 18:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 18:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 18:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 18:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 18:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 18:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 18:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 18:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 18:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 18:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 18:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 18:53:09,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 18:53:09,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 18:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 18:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 18:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 18:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 18:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 18:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 18:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 18:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 18:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 18:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 18:53:09,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 18:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 18:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 18:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 18:53:09,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 18:53:10,099 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03825244804935104, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057536403349859384, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03825244804935104, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17141666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.235, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 18:53:10,099 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 18:53:10,100 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17592210743941591, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03825244804935104, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057536403349859384, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03825244804935104, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17141666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.235, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 18:53:10,485 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 525 ms to try 5000 different iterations. Found a solution that moves 2575 regions; Going from a computed imbalance of 0.17592210743941591 to a new imbalance of 0.015585864712666637. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018609205557672703); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03354478387547794, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2861111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018609205557672703); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043583333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 18:53:10,498 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv886561683:77 , srv874483997:79 , srv1666719485:80 , srv1165743802:81 , srv1263413818:81 , srv485793388:81 , srv710191511:81 , srv1127037739:82 , srv1580694600:82 , srv2038649829:82 , srv29173197:82 , srv1030835280:83 , srv2040070986:83 , srv312142622:83 , srv356035052:83 , srv726185941:83 , srv915739005:83 , srv1027925338:84 , srv1064131452:84 , srv1342719351:84 , srv1479418491:84 , srv1522921333:84 , srv1764544001:84 , srv1907038284:84 , srv2006142720:84 , srv2025243316:84 , srv2040188673:84 , srv449749423:84 , srv56451041:84 , srv74120033:84 , srv776935265:84 , srv84005535:84 , srv846127248:84 , srv924461691:84 , srv1120722448:85 , srv1492995673:85 , srv1552738461:85 , srv1601350001:85 , srv1833267415:85 , srv1855183840:85 , srv608913086:85 , srv839592942:85 , srv1042179257:86 , srv1125781138:86 , srv123551959:86 , srv150575840:86 , srv1516218974:86 , srv1701171541:86 , srv1743740137:86 , srv175915686:86 , srv1845423433:86 , srv2057462117:86 , srv2145535204:86 , srv628407792:86 , srv936459373:86 , srv1135179152:87 , srv1184758474:87 , srv1190852800:87 , srv1274420825:87 , srv1360598204:87 , srv1495242089:87 , srv1569119757:87 , srv1587626392:87 , srv16620023:87 , srv1662233129:87 , srv1803164870:87 , srv1857574904:87 , srv1963137198:87 , srv2000998320:87 , srv2004367761:87 , srv562271760:87 , srv69062308:87 , srv1006627613:88 , srv1036285527:88 , srv1191212164:88 , srv1345370869:88 , srv1677414980:88 , srv1781448719:88 , srv1963987996:88 , srv2083170357:88 , srv25186063:88 , srv275649812:88 , srv297757912:88 , srv418500352:88 , srv873603612:88 , srv1099488219:89 , srv1226047727:89 , srv1423676235:89 , srv1467085638:89 , srv1886914667:89 , srv2069926856:89 , srv301788563:89 , srv321789151:89 , srv347924654:89 , srv891905771:89 , srv1521119703:90 , srv395334574:90 , srv89525353:92 , srv1297321800:94 , srv193968837:509 } 2024-02-11 18:53:10,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 18:53:10,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1030835280=2, srv1963987996=55, srv321789151=73, srv1342719351=20, srv2006142720=58, srv1521119703=30, srv1099488219=6, srv1803164870=46, srv1190852800=13, srv2069926856=64, srv1495242089=27, srv1027925338=1, srv710191511=85, srv924461691=98, srv1263413818=17, srv1601350001=36, srv1907038284=52, srv1855183840=49, srv915739005=97, srv69062308=84, srv1552738461=32, srv1587626392=35, srv2040070986=61, srv874483997=93, srv1833267415=47, srv193968837=53, srv2025243316=59, srv1677414980=40, srv1042179257=4, srv1701171541=41, srv301788563=71, srv1467085638=24, srv1479418491=25, srv1423676235=23, srv297757912=70, srv2004367761=57, srv2083170357=65, srv84005535=90, srv936459373=99, srv1781448719=45, srv175915686=43, srv56451041=81, srv846127248=91, srv312142622=72, srv891905771=95, srv1184758474=12, srv1857574904=50, srv1963137198=54, srv1127037739=9, srv395334574=76, srv886561683=94, srv562271760=80, srv1662233129=38, srv1125781138=8, srv1516218974=29, srv1297321800=19, srv1845423433=48, srv2000998320=56, srv74120033=87, srv150575840=28, srv1743740137=42, srv1036285527=3, srv2038649829=60, srv628407792=83, srv1522921333=31, srv2145535204=66, srv726185941=86, srv1191212164=14, srv2040188673=62, srv275649812=68, srv1274420825=18, srv1064131452=5, srv776935265=88, srv608913086=82, srv2057462117=63, srv1120722448=7, srv29173197=69, srv1006627613=0, srv1345370869=21, srv1165743802=11, srv356035052=75, srv1666719485=39, srv89525353=96, srv1135179152=10, srv1360598204=22, srv1492995673=26, srv123551959=16, srv347924654=74, srv449749423=78, srv873603612=92, srv1580694600=34, srv839592942=89, srv25186063=67, srv1569119757=33, srv418500352=77, srv1764544001=44, srv16620023=37, srv485793388=79, srv1226047727=15, srv1886914667=51} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 18:53:10,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 18:53:10,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 18:53:10,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 18:53:10,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 18:53:10,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 18:53:10,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 18:53:10,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 18:53:10,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 18:53:10,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 18:53:10,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 18:53:10,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 18:53:10,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 18:53:10,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 18:53:10,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 18:53:10,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 18:53:10,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 18:53:10,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 18:53:10,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 18:53:10,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 18:53:10,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 18:53:10,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 18:53:10,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 18:53:10,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 18:53:10,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 18:53:10,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 18:53:10,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 18:53:10,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 18:53:10,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 18:53:10,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 18:53:10,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 18:53:10,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 18:53:10,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 18:53:10,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 18:53:10,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 18:53:10,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 18:53:10,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 18:53:10,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 18:53:10,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 18:53:10,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 18:53:10,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 18:53:10,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 18:53:10,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 18:53:10,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 18:53:10,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 18:53:10,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 18:53:10,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 18:53:10,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 18:53:10,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 18:53:10,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 18:53:10,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 18:53:10,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 18:53:10,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 18:53:10,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 18:53:10,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 18:53:10,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 18:53:10,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 18:53:10,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 18:53:10,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 18:53:10,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 18:53:10,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 18:53:10,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 18:53:10,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 18:53:10,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 18:53:10,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 18:53:10,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 18:53:10,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 18:53:10,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 18:53:10,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 18:53:10,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 18:53:10,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 18:53:10,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 18:53:10,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 18:53:10,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 18:53:10,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 18:53:10,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 18:53:10,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 18:53:10,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 18:53:10,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 18:53:10,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 18:53:10,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 18:53:10,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 18:53:10,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 18:53:10,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 18:53:10,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 18:53:10,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 18:53:10,693 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018609205557672703); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03354478387547794, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018609205557672703); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043583333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 18:53:10,694 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 18:53:10,694 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015567834177292272, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018609205557672703); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03354478387547794, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018609205557672703); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043583333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 18:53:11,030 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 462 ms to try 5000 different iterations. Found a solution that moves 1017 regions; Going from a computed imbalance of 0.015567834177292272 to a new imbalance of 5.043722223382227E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004984074901000224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006800013203921953); MoveCostFunction : (multiplier=7.0, imbalance=0.113, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004984074901000224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00325); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 18:53:11,032 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1297321800:78 , srv874483997:81 , srv1190852800:82 , srv1677414980:82 , srv1781448719:82 , srv25186063:82 , srv89525353:82 , srv1006627613:83 , srv1099488219:83 , srv1467085638:83 , srv1521119703:83 , srv1845423433:83 , srv2069926856:83 , srv449749423:83 , srv1027925338:84 , srv1345370869:84 , srv1701171541:84 , srv1803164870:84 , srv2000998320:84 , srv297757912:84 , srv395334574:84 , srv418500352:84 , srv562271760:84 , srv936459373:84 , srv1036285527:85 , srv1125781138:85 , srv1184758474:85 , srv1191212164:85 , srv1226047727:85 , srv123551959:85 , srv1274420825:85 , srv1423676235:85 , srv1495242089:85 , srv1516218974:85 , srv1552738461:85 , srv1601350001:85 , srv16620023:85 , srv1743740137:85 , srv1855183840:85 , srv2004367761:85 , srv2025243316:85 , srv2040188673:85 , srv275649812:85 , srv29173197:85 , srv301788563:85 , srv347924654:85 , srv485793388:85 , srv628407792:85 , srv74120033:85 , srv776935265:85 , srv839592942:85 , srv84005535:85 , srv873603612:85 , srv1064131452:86 , srv1120722448:86 , srv1127037739:86 , srv1135179152:86 , srv1360598204:86 , srv1580694600:86 , srv1587626392:86 , srv1666719485:86 , srv175915686:86 , srv1833267415:86 , srv1857574904:86 , srv1886914667:86 , srv1907038284:86 , srv1963137198:86 , srv1963987996:86 , srv2006142720:86 , srv2038649829:86 , srv2057462117:86 , srv321789151:86 , srv56451041:86 , srv69062308:86 , srv710191511:86 , srv726185941:86 , srv846127248:86 , srv915739005:86 , srv924461691:86 , srv1030835280:87 , srv1165743802:87 , srv1263413818:87 , srv1342719351:87 , srv1479418491:87 , srv1492995673:87 , srv150575840:87 , srv1522921333:87 , srv1662233129:87 , srv1764544001:87 , srv2083170357:87 , srv2145535204:87 , srv312142622:87 , srv356035052:87 , srv891905771:87 , srv1042179257:88 , srv1569119757:88 , srv608913086:88 , srv886561683:88 , srv2040070986:89 , srv193968837:563 } 2024-02-11 18:53:11,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 18:53:11,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1030835280=2, srv1963987996=55, srv321789151=73, srv1342719351=20, srv2006142720=58, srv1521119703=30, srv1099488219=6, srv1803164870=46, srv1190852800=13, srv2069926856=64, srv1495242089=27, srv1027925338=1, srv710191511=85, srv924461691=98, srv1263413818=17, srv1601350001=36, srv1907038284=52, srv1855183840=49, srv915739005=97, srv69062308=84, srv1552738461=32, srv1587626392=35, srv2040070986=61, srv874483997=93, srv1833267415=47, srv193968837=53, srv2025243316=59, srv1677414980=40, srv1042179257=4, srv1701171541=41, srv301788563=71, srv1467085638=24, srv1479418491=25, srv1423676235=23, srv297757912=70, srv2004367761=57, srv2083170357=65, srv84005535=90, srv936459373=99, srv1781448719=45, srv175915686=43, srv56451041=81, srv846127248=91, srv312142622=72, srv891905771=95, srv1184758474=12, srv1857574904=50, srv1963137198=54, srv1127037739=9, srv395334574=76, srv886561683=94, srv562271760=80, srv1662233129=38, srv1125781138=8, srv1516218974=29, srv1297321800=19, srv1845423433=48, srv2000998320=56, srv74120033=87, srv150575840=28, srv1743740137=42, srv1036285527=3, srv2038649829=60, srv628407792=83, srv1522921333=31, srv2145535204=66, srv726185941=86, srv1191212164=14, srv2040188673=62, srv275649812=68, srv1274420825=18, srv1064131452=5, srv776935265=88, srv608913086=82, srv2057462117=63, srv1120722448=7, srv29173197=69, srv1006627613=0, srv1345370869=21, srv1165743802=11, srv356035052=75, srv1666719485=39, srv89525353=96, srv1135179152=10, srv1360598204=22, srv1492995673=26, srv123551959=16, srv347924654=74, srv449749423=78, srv873603612=92, srv1580694600=34, srv839592942=89, srv25186063=67, srv1569119757=33, srv418500352=77, srv1764544001=44, srv16620023=37, srv485793388=79, srv1226047727=15, srv1886914667=51} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 18:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 18:53:11,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 18:53:11,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 18:53:11,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 18:53:11,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 18:53:11,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 18:53:11,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 18:53:11,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 18:53:11,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 18:53:11,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 18:53:11,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 18:53:11,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 18:53:11,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 18:53:11,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 18:53:11,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 18:53:11,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 18:53:11,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 18:53:11,152 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004984074901000224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006800013203921953); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004984074901000224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00325); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 18:53:11,152 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 18:53:11,153 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.972510361340581E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004984074901000224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006800013203921953); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004984074901000224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00325); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 18:53:11,356 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 296 ms to try 5000 different iterations. Found a solution that moves 226 regions; Going from a computed imbalance of 4.972510361340581E-4 to a new imbalance of 2.0283363898751138E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.025111111111111112, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 18:53:11,357 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2040070986:82 , srv1492995673:83 , srv1886914667:83 , srv710191511:83 , srv839592942:83 , srv891905771:83 , srv1027925338:84 , srv1030835280:84 , srv1064131452:84 , srv1099488219:84 , srv1127037739:84 , srv1135179152:84 , srv1184758474:84 , srv1191212164:84 , srv1226047727:84 , srv1263413818:84 , srv1274420825:84 , srv1342719351:84 , srv1345370869:84 , srv1360598204:84 , srv1479418491:84 , srv150575840:84 , srv1522921333:84 , srv1569119757:84 , srv1587626392:84 , srv1601350001:84 , srv16620023:84 , srv1662233129:84 , srv1701171541:84 , srv1743740137:84 , srv1833267415:84 , srv1855183840:84 , srv1857574904:84 , srv2006142720:84 , srv2040188673:84 , srv2069926856:84 , srv2083170357:84 , srv25186063:84 , srv301788563:84 , srv312142622:84 , srv347924654:84 , srv356035052:84 , srv418500352:84 , srv562271760:84 , srv608913086:84 , srv628407792:84 , srv74120033:84 , srv873603612:84 , srv874483997:84 , srv89525353:84 , srv924461691:84 , srv936459373:84 , srv1006627613:85 , srv1036285527:85 , srv1042179257:85 , srv1120722448:85 , srv1125781138:85 , srv1165743802:85 , srv1190852800:85 , srv123551959:85 , srv1297321800:85 , srv1423676235:85 , srv1467085638:85 , srv1495242089:85 , srv1516218974:85 , srv1521119703:85 , srv1552738461:85 , srv1580694600:85 , srv1666719485:85 , srv1677414980:85 , srv175915686:85 , srv1781448719:85 , srv1845423433:85 , srv1907038284:85 , srv1963137198:85 , srv1963987996:85 , srv2004367761:85 , srv2025243316:85 , srv2038649829:85 , srv2057462117:85 , srv2145535204:85 , srv275649812:85 , srv29173197:85 , srv297757912:85 , srv321789151:85 , srv395334574:85 , srv485793388:85 , srv56451041:85 , srv69062308:85 , srv726185941:85 , srv776935265:85 , srv84005535:85 , srv846127248:85 , srv886561683:85 , srv915739005:85 , srv1764544001:86 , srv1803164870:86 , srv2000998320:86 , srv449749423:86 , srv193968837:640 } 2024-02-11 18:53:11,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 18:53:11,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1030835280=2, srv1963987996=55, srv321789151=73, srv1342719351=20, srv2006142720=58, srv1521119703=30, srv1099488219=6, srv1803164870=46, srv1190852800=13, srv2069926856=64, srv1495242089=27, srv1027925338=1, srv710191511=85, srv924461691=98, srv1263413818=17, srv1601350001=36, srv1907038284=52, srv1855183840=49, srv915739005=97, srv69062308=84, srv1552738461=32, srv1587626392=35, srv2040070986=61, srv874483997=93, srv1833267415=47, srv193968837=53, srv2025243316=59, srv1677414980=40, srv1042179257=4, srv1701171541=41, srv301788563=71, srv1467085638=24, srv1479418491=25, srv1423676235=23, srv297757912=70, srv2004367761=57, srv2083170357=65, srv84005535=90, srv936459373=99, srv1781448719=45, srv175915686=43, srv56451041=81, srv846127248=91, srv312142622=72, srv891905771=95, srv1184758474=12, srv1857574904=50, srv1963137198=54, srv1127037739=9, srv395334574=76, srv886561683=94, srv562271760=80, srv1662233129=38, srv1125781138=8, srv1516218974=29, srv1297321800=19, srv1845423433=48, srv2000998320=56, srv74120033=87, srv150575840=28, srv1743740137=42, srv1036285527=3, srv2038649829=60, srv628407792=83, srv1522921333=31, srv2145535204=66, srv726185941=86, srv1191212164=14, srv2040188673=62, srv275649812=68, srv1274420825=18, srv1064131452=5, srv776935265=88, srv608913086=82, srv2057462117=63, srv1120722448=7, srv29173197=69, srv1006627613=0, srv1345370869=21, srv1165743802=11, srv356035052=75, srv1666719485=39, srv89525353=96, srv1135179152=10, srv1360598204=22, srv1492995673=26, srv123551959=16, srv347924654=74, srv449749423=78, srv873603612=92, srv1580694600=34, srv839592942=89, srv25186063=67, srv1569119757=33, srv418500352=77, srv1764544001=44, srv16620023=37, srv485793388=79, srv1226047727=15, srv1886914667=51} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 18:53:11,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 18:53:11,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 18:53:11,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 18:53:11,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 18:53:11,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 18:53:11,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 18:53:11,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 18:53:11,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 18:53:11,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 18:53:11,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 18:53:11,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 18:53:11,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 18:53:11,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 18:53:11,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 18:53:11,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 18:53:11,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 18:53:11,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 18:53:11,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 18:53:11,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 18:53:11,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 18:53:11,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 18:53:11,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 18:53:11,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 18:53:11,472 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.870087807560343E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 18:53:11,472 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv2040070986:82 , srv1492995673:83 , srv1886914667:83 , srv710191511:83 , srv839592942:83 , srv891905771:83 , srv1027925338:84 , srv1030835280:84 , srv1064131452:84 , srv1099488219:84 , srv1127037739:84 , srv1135179152:84 , srv1184758474:84 , srv1191212164:84 , srv1226047727:84 , srv1263413818:84 , srv1274420825:84 , srv1342719351:84 , srv1345370869:84 , srv1360598204:84 , srv1479418491:84 , srv150575840:84 , srv1522921333:84 , srv1569119757:84 , srv1587626392:84 , srv1601350001:84 , srv16620023:84 , srv1662233129:84 , srv1701171541:84 , srv1743740137:84 , srv1833267415:84 , srv1855183840:84 , srv1857574904:84 , srv2006142720:84 , srv2040188673:84 , srv2069926856:84 , srv2083170357:84 , srv25186063:84 , srv301788563:84 , srv312142622:84 , srv347924654:84 , srv356035052:84 , srv418500352:84 , srv562271760:84 , srv608913086:84 , srv628407792:84 , srv74120033:84 , srv873603612:84 , srv874483997:84 , srv89525353:84 , srv924461691:84 , srv936459373:84 , srv1006627613:85 , srv1036285527:85 , srv1042179257:85 , srv1120722448:85 , srv1125781138:85 , srv1165743802:85 , srv1190852800:85 , srv123551959:85 , srv1297321800:85 , srv1423676235:85 , srv1467085638:85 , srv1495242089:85 , srv1516218974:85 , srv1521119703:85 , srv1552738461:85 , srv1580694600:85 , srv1666719485:85 , srv1677414980:85 , srv175915686:85 , srv1781448719:85 , srv1845423433:85 , srv1907038284:85 , srv1963137198:85 , srv1963987996:85 , srv2004367761:85 , srv2025243316:85 , srv2038649829:85 , srv2057462117:85 , srv2145535204:85 , srv275649812:85 , srv29173197:85 , srv297757912:85 , srv321789151:85 , srv395334574:85 , srv485793388:85 , srv56451041:85 , srv69062308:85 , srv726185941:85 , srv776935265:85 , srv84005535:85 , srv846127248:85 , srv886561683:85 , srv915739005:85 , srv1764544001:86 , srv1803164870:86 , srv2000998320:86 , srv449749423:86 , srv193968837:640 } 2024-02-11 18:53:11,499 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=281 (was 281), ProcessCount=168 (was 168), AvailableMemoryMB=2528 (was 3291) 2024-02-11 18:53:11,506 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=281, ProcessCount=168, AvailableMemoryMB=2528 2024-02-11 18:53:11,506 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-11 18:53:11,507 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-02-11 18:53:11,516 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1098104838:84 , srv1124540611:84 , srv1165990160:84 , srv1334244189:84 , srv1543633894:84 , srv1645869905:84 , srv1672640790:84 , srv1711487703:84 , srv1875082963:84 , srv1892918991:84 , srv1963017970:84 , srv2011387312:84 , srv2020623907:84 , srv2101201596:84 , srv2126270617:84 , srv475710976:84 , srv577996999:84 , srv633070395:84 , srv642794674:84 , srv749018447:84 , srv78694156:84 , srv799845282:84 , srv801454847:84 , srv806412969:84 , srv890615544:84 , srv922127510:84 , srv927472945:84 , srv938470866:84 , srv972878899:84 , srv2050442461:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-02-11 18:53:11,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 18:53:11,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1165990160=2, srv1334244189=3, srv1875082963=8, srv1672640790=6, srv577996999=17, srv2050442461=13, srv633070395=18, srv749018447=20, srv799845282=22, srv1543633894=4, srv2011387312=11, srv972878899=29, srv2101201596=14, srv938470866=28, srv1711487703=7, srv1098104838=0, srv78694156=21, srv2126270617=15, srv890615544=25, srv922127510=26, srv1963017970=10, srv642794674=19, srv806412969=24, srv475710976=16, srv1645869905=5, srv801454847=23, srv2020623907=12, srv1124540611=1, srv927472945=27, srv1892918991=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 18:53:11,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 18:53:11,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 18:53:11,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 18:53:11,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 18:53:11,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 18:53:11,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 18:53:11,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 18:53:11,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 18:53:11,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 18:53:11,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-11 18:53:11,545 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.191828105226989, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 18:53:11,545 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912797487360085, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.191828105226989, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-02-11 18:55:11,533 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 7319386 different iterations. Found a solution that moves 1849 regions; Going from a computed imbalance of 0.9912797487360085 to a new imbalance of 4.3156582404131396E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6848148148148148, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 18:55:11,540 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1098104838:90 , srv1124540611:90 , srv1165990160:90 , srv1334244189:90 , srv1543633894:90 , srv1645869905:90 , srv1672640790:90 , srv1711487703:90 , srv1875082963:90 , srv1892918991:90 , srv1963017970:90 , srv2011387312:90 , srv2020623907:90 , srv2050442461:90 , srv2101201596:90 , srv2126270617:90 , srv475710976:90 , srv577996999:90 , srv633070395:90 , srv642794674:90 , srv749018447:90 , srv78694156:90 , srv799845282:90 , srv801454847:90 , srv806412969:90 , srv890615544:90 , srv922127510:90 , srv927472945:90 , srv938470866:90 , srv972878899:90 } 2024-02-11 18:55:11,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 18:55:11,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1165990160=2, srv1334244189=3, srv1875082963=8, srv1672640790=6, srv577996999=17, srv2050442461=13, srv633070395=18, srv749018447=20, srv799845282=22, srv1543633894=4, srv2011387312=11, srv972878899=29, srv2101201596=14, srv938470866=28, srv1711487703=7, srv1098104838=0, srv78694156=21, srv2126270617=15, srv890615544=25, srv922127510=26, srv1963017970=10, srv642794674=19, srv806412969=24, srv475710976=16, srv1645869905=5, srv801454847=23, srv2020623907=12, srv1124540611=1, srv927472945=27, srv1892918991=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 18:55:11,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-11 18:55:11,558 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 18:55:11,558 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1098104838:90 , srv1124540611:90 , srv1165990160:90 , srv1334244189:90 , srv1543633894:90 , srv1645869905:90 , srv1672640790:90 , srv1711487703:90 , srv1875082963:90 , srv1892918991:90 , srv1963017970:90 , srv2011387312:90 , srv2020623907:90 , srv2050442461:90 , srv2101201596:90 , srv2126270617:90 , srv475710976:90 , srv577996999:90 , srv633070395:90 , srv642794674:90 , srv749018447:90 , srv78694156:90 , srv799845282:90 , srv801454847:90 , srv806412969:90 , srv890615544:90 , srv922127510:90 , srv927472945:90 , srv938470866:90 , srv972878899:90 } 2024-02-11 18:55:11,570 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=233 (was 281), ProcessCount=167 (was 168), AvailableMemoryMB=2081 (was 2528)