2023-02-22 10:54:02,295 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-02-22 10:54:02,632 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-22 10:54:02,756 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-22 10:54:02,814 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-22 10:54:02,823 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-22 10:54:02,836 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=272, ProcessCount=172, AvailableMemoryMB=8308 2023-02-22 10:54:02,839 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-22 10:54:02,839 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-22 10:54:02,966 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010996446:84 , srv1073316192:84 , srv1084181127:84 , srv1086155883:84 , srv1098140890:84 , srv1106367621:84 , srv1106969418:84 , srv1146564879:84 , srv1152671314:84 , srv1165094100:84 , srv1177374108:84 , srv1189579352:84 , srv1212461688:84 , srv123770847:84 , srv1244425057:84 , srv1286650195:84 , srv1298350669:84 , srv1321661093:84 , srv132197638:84 , srv134655009:84 , srv1377663047:84 , srv1399861016:84 , srv1411515675:84 , srv1412032973:84 , srv1421239077:84 , srv1431245727:84 , srv1458090525:84 , srv1464736648:84 , srv154091362:84 , srv1613618982:84 , srv1634015851:84 , srv1636343897:84 , srv1642612350:84 , srv1680364082:84 , srv1688602056:84 , srv1704090665:84 , srv1709075086:84 , srv1709530603:84 , srv1753623144:84 , srv1806642825:84 , srv1819006658:84 , srv1825475353:84 , srv1857306225:84 , srv1871253989:84 , srv1879951075:84 , srv1907254375:84 , srv1908437452:84 , srv1912110496:84 , srv1919827381:84 , srv1929332035:84 , srv197156795:84 , srv2044750023:84 , srv2058447962:84 , srv2069860156:84 , srv2101955005:84 , srv2136583956:84 , srv224069938:84 , srv226104022:84 , srv240363101:84 , srv246339268:84 , srv280976307:84 , srv286853314:84 , srv292695006:84 , srv29975801:84 , srv30763917:84 , srv30992519:84 , srv320771678:84 , srv333257660:84 , srv351514047:84 , srv369233892:84 , srv373058627:84 , srv381842997:84 , srv392489075:84 , srv460146061:84 , srv492983775:84 , srv544264465:84 , srv551061371:84 , srv560876083:84 , srv5854385:84 , srv630002184:84 , srv652141122:84 , srv653161803:84 , srv671361280:84 , srv673413388:84 , srv68709965:84 , srv727465278:84 , srv729595772:84 , srv761262433:84 , srv805050468:84 , srv830548871:84 , srv833017864:84 , srv885983827:84 , srv901324713:84 , srv913308369:84 , srv919822873:84 , srv938125299:84 , srv958372030:84 , srv964706086:84 , srv980211303:84 , srv1969719604:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-02-22 10:54:02,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-22 10:54:02,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv68709965=85, srv246339268=60, srv560876083=78, srv392489075=73, srv1753623144=38, srv373058627=71, srv381842997=72, srv30763917=65, srv1688602056=34, srv727465278=86, srv280976307=61, srv1857306225=42, srv1010996446=0, srv1912110496=47, srv1919827381=48, srv761262433=88, srv333257660=68, srv1907254375=45, srv1286650195=15, srv1244425057=14, srv1458090525=26, srv980211303=99, srv1709530603=37, srv1613618982=29, srv1929332035=49, srv671361280=83, srv1908437452=46, srv1084181127=2, srv2058447962=53, srv1165094100=9, srv1879951075=44, srv460146061=74, srv830548871=90, srv1871253989=43, srv653161803=82, srv1106969418=6, srv1152671314=8, srv351514047=69, srv919822873=95, srv964706086=98, srv1177374108=10, srv1704090665=35, srv1431245727=25, srv551061371=77, srv1086155883=3, srv123770847=13, srv1106367621=5, srv226104022=58, srv154091362=28, srv369233892=70, srv1819006658=40, srv240363101=59, srv833017864=91, srv1969719604=50, srv1212461688=12, srv938125299=96, srv1412032973=23, srv2136583956=56, srv29975801=64, srv1098140890=4, srv132197638=18, srv492983775=75, srv2069860156=54, srv5854385=79, srv197156795=51, srv1709075086=36, srv630002184=80, srv544264465=76, srv885983827=92, srv1298350669=16, srv1642612350=32, srv1146564879=7, srv1377663047=20, srv134655009=19, srv320771678=67, srv292695006=63, srv901324713=93, srv1321661093=17, srv1680364082=33, srv1634015851=30, srv1825475353=41, srv1421239077=24, srv1189579352=11, srv729595772=87, srv2101955005=55, srv286853314=62, srv958372030=97, srv30992519=66, srv1806642825=39, srv1399861016=21, srv1464736648=27, srv913308369=94, srv652141122=81, srv1411515675=22, srv1636343897=31, srv673413388=84, srv1073316192=1, srv805050468=89, srv2044750023=52, srv224069938=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-22 10:54:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-22 10:54:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-22 10:54:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-22 10:54:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-22 10:54:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-22 10:54:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-22 10:54:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-22 10:54:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-22 10:54:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-22 10:54:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-22 10:54:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-22 10:54:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-22 10:54:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-22 10:54:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-22 10:54:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-22 10:54:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-22 10:54:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-22 10:54:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-22 10:54:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-22 10:54:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-22 10:54:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-22 10:54:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-22 10:54:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-22 10:54:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-22 10:54:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-22 10:54:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-22 10:54:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-22 10:54:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-22 10:54:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-22 10:54:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-22 10:54:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-22 10:54:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-22 10:54:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-22 10:54:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-22 10:54:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-22 10:54:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-22 10:54:03,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-22 10:54:03,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-22 10:54:03,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-22 10:54:03,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-22 10:54:03,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-22 10:54:03,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-22 10:54:03,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-22 10:54:03,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-22 10:54:03,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-22 10:54:03,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-22 10:54:03,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-22 10:54:03,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-22 10:54:03,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-22 10:54:03,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-22 10:54:03,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-22 10:54:03,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-22 10:54:03,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-22 10:54:03,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-22 10:54:03,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-22 10:54:03,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-22 10:54:03,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-22 10:54:03,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-22 10:54:03,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-22 10:54:03,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-22 10:54:03,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-22 10:54:03,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-22 10:54:03,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-22 10:54:03,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-22 10:54:03,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-22 10:54:03,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-22 10:54:03,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-22 10:54:03,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-22 10:54:03,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-22 10:54:03,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-22 10:54:03,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-22 10:54:03,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-22 10:54:03,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-22 10:54:03,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-22 10:54:03,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-22 10:54:03,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-22 10:54:03,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-22 10:54:03,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-22 10:54:03,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-22 10:54:03,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-22 10:54:03,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-22 10:54:03,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-22 10:54:03,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-22 10:54:03,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-22 10:54:03,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-22 10:54:03,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-22 10:54:03,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-22 10:54:03,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-22 10:54:03,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-22 10:54:03,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-22 10:54:03,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-22 10:54:03,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-22 10:54:03,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-22 10:54:03,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-22 10:54:03,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-22 10:54:03,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-22 10:54:03,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-22 10:54:03,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-22 10:54:03,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-22 10:54:03,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-22 10:54:03,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-22 10:54:03,373 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:54:03,373 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-22 10:54:03,374 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-22 10:54:04,040 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 943 ms to try 5000 different iterations. Found a solution that moves 4695 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18154956697596364. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.042474290397911126, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06087834458578989, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5216666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.042474290397911126, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.177, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24091666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:54:04,061 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1189579352:81 , srv2044750023:81 , srv224069938:81 , srv392489075:81 , srv805050468:81 , srv1298350669:82 , srv1377663047:82 , srv1680364082:82 , srv2058447962:82 , srv351514047:82 , srv492983775:82 , srv1010996446:83 , srv1165094100:83 , srv1177374108:83 , srv1912110496:83 , srv240363101:83 , srv30763917:83 , srv630002184:83 , srv727465278:83 , srv761262433:83 , srv1321661093:84 , srv1642612350:84 , srv1709075086:84 , srv1709530603:84 , srv1819006658:84 , srv197156795:84 , srv2136583956:84 , srv30992519:84 , srv333257660:84 , srv551061371:84 , srv1084181127:85 , srv1152671314:85 , srv1286650195:85 , srv1421239077:85 , srv1464736648:85 , srv154091362:85 , srv1753623144:85 , srv1806642825:85 , srv1825475353:85 , srv1879951075:85 , srv1907254375:85 , srv1929332035:85 , srv226104022:85 , srv320771678:85 , srv5854385:85 , srv652141122:85 , srv671361280:85 , srv958372030:85 , srv1106367621:86 , srv1212461688:86 , srv1613618982:86 , srv1636343897:86 , srv246339268:86 , srv292695006:86 , srv369233892:86 , srv373058627:86 , srv544264465:86 , srv729595772:86 , srv901324713:86 , srv919822873:86 , srv938125299:86 , srv980211303:86 , srv1073316192:87 , srv1146564879:87 , srv134655009:87 , srv1704090665:87 , srv1857306225:87 , srv1919827381:87 , srv673413388:87 , srv885983827:87 , srv1086155883:88 , srv1106969418:88 , srv123770847:88 , srv1399861016:88 , srv1634015851:88 , srv286853314:88 , srv29975801:88 , srv68709965:88 , srv132197638:89 , srv1412032973:89 , srv1871253989:89 , srv280976307:89 , srv381842997:89 , srv913308369:89 , srv964706086:89 , srv1098140890:90 , srv1244425057:90 , srv1908437452:90 , srv653161803:90 , srv1431245727:91 , srv1458090525:91 , srv2101955005:91 , srv830548871:91 , srv833017864:91 , srv1411515675:93 , srv560876083:94 , srv2069860156:96 , srv1688602056:97 , srv460146061:97 , srv1969719604:467 } 2023-02-22 10:54:04,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-22 10:54:04,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv68709965=85, srv246339268=60, srv560876083=78, srv392489075=73, srv1753623144=38, srv373058627=71, srv381842997=72, srv30763917=65, srv1688602056=34, srv727465278=86, srv280976307=61, srv1857306225=42, srv1010996446=0, srv1912110496=47, srv1919827381=48, srv761262433=88, srv333257660=68, srv1907254375=45, srv1286650195=15, srv1244425057=14, srv1458090525=26, srv980211303=99, srv1709530603=37, srv1613618982=29, srv1929332035=49, srv671361280=83, srv1908437452=46, srv1084181127=2, srv2058447962=53, srv1165094100=9, srv1879951075=44, srv460146061=74, srv830548871=90, srv1871253989=43, srv653161803=82, srv1106969418=6, srv1152671314=8, srv351514047=69, srv919822873=95, srv964706086=98, srv1177374108=10, srv1704090665=35, srv1431245727=25, srv551061371=77, srv1086155883=3, srv123770847=13, srv1106367621=5, srv226104022=58, srv154091362=28, srv369233892=70, srv1819006658=40, srv240363101=59, srv833017864=91, srv1969719604=50, srv1212461688=12, srv938125299=96, srv1412032973=23, srv2136583956=56, srv29975801=64, srv1098140890=4, srv132197638=18, srv492983775=75, srv2069860156=54, srv5854385=79, srv197156795=51, srv1709075086=36, srv630002184=80, srv544264465=76, srv885983827=92, srv1298350669=16, srv1642612350=32, srv1146564879=7, srv1377663047=20, srv134655009=19, srv320771678=67, srv292695006=63, srv901324713=93, srv1321661093=17, srv1680364082=33, srv1634015851=30, srv1825475353=41, srv1421239077=24, srv1189579352=11, srv729595772=87, srv2101955005=55, srv286853314=62, srv958372030=97, srv30992519=66, srv1806642825=39, srv1399861016=21, srv1464736648=27, srv913308369=94, srv652141122=81, srv1411515675=22, srv1636343897=31, srv673413388=84, srv1073316192=1, srv805050468=89, srv2044750023=52, srv224069938=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-22 10:54:04,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-22 10:54:04,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-22 10:54:04,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-22 10:54:04,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-22 10:54:04,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-22 10:54:04,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-22 10:54:04,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-22 10:54:04,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-22 10:54:04,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-22 10:54:04,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-22 10:54:04,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-22 10:54:04,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-22 10:54:04,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-22 10:54:04,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-22 10:54:04,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-22 10:54:04,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-22 10:54:04,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-22 10:54:04,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-22 10:54:04,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-22 10:54:04,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-22 10:54:04,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-22 10:54:04,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-22 10:54:04,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-22 10:54:04,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-22 10:54:04,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-22 10:54:04,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-22 10:54:04,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-22 10:54:04,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-22 10:54:04,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-22 10:54:04,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-22 10:54:04,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-22 10:54:04,320 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.042474290397911126, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06087834458578989, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.042474290397911126, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.177, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24091666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:54:04,320 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-22 10:54:04,321 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18151669188331018, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.042474290397911126, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06087834458578989, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.042474290397911126, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.177, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24091666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-22 10:54:04,657 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 553 ms to try 5000 different iterations. Found a solution that moves 2583 regions; Going from a computed imbalance of 0.18151669188331018 to a new imbalance of 0.01837633701783493. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02340137585845833); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039985968358217015, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.287, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02340137585845833); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.015083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.049833333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:54:04,668 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv460146061:74 , srv1688602056:77 , srv2069860156:77 , srv560876083:78 , srv1458090525:79 , srv2101955005:79 , srv1244425057:80 , srv1411515675:80 , srv830548871:80 , srv833017864:80 , srv1431245727:81 , srv913308369:81 , srv1086155883:82 , srv1098140890:82 , srv132197638:82 , srv286853314:82 , srv653161803:82 , srv964706086:82 , srv1146564879:83 , srv1412032973:83 , srv1908437452:83 , srv1084181127:84 , srv1106969418:84 , srv1399861016:84 , srv1613618982:84 , srv1634015851:84 , srv1636343897:84 , srv1704090665:84 , srv1825475353:84 , srv246339268:84 , srv280976307:84 , srv381842997:84 , srv68709965:84 , srv1212461688:85 , srv134655009:85 , srv1464736648:85 , srv154091362:85 , srv1857306225:85 , srv1919827381:85 , srv2136583956:85 , srv29975801:85 , srv373058627:85 , srv544264465:85 , srv673413388:85 , srv919822873:85 , srv958372030:85 , srv1106367621:86 , srv1152671314:86 , srv123770847:86 , srv1286650195:86 , srv1753623144:86 , srv1871253989:86 , srv1879951075:86 , srv369233892:86 , srv727465278:86 , srv729595772:86 , srv1073316192:87 , srv1421239077:87 , srv1709075086:87 , srv292695006:87 , srv30763917:87 , srv320771678:87 , srv333257660:87 , srv671361280:87 , srv885983827:87 , srv901324713:87 , srv1709530603:88 , srv1806642825:88 , srv1929332035:88 , srv492983775:88 , srv551061371:88 , srv5854385:88 , srv630002184:88 , srv652141122:88 , srv938125299:88 , srv980211303:88 , srv1321661093:89 , srv1377663047:89 , srv1642612350:89 , srv1819006658:89 , srv197156795:89 , srv226104022:89 , srv240363101:89 , srv30992519:89 , srv761262433:89 , srv1189579352:90 , srv1907254375:90 , srv1912110496:90 , srv2058447962:90 , srv224069938:90 , srv1165094100:91 , srv1177374108:91 , srv1680364082:91 , srv2044750023:91 , srv805050468:91 , srv1298350669:92 , srv351514047:92 , srv392489075:92 , srv1010996446:93 , srv1969719604:515 } 2023-02-22 10:54:04,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-22 10:54:04,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv68709965=85, srv246339268=60, srv560876083=78, srv392489075=73, srv1753623144=38, srv373058627=71, srv381842997=72, srv30763917=65, srv1688602056=34, srv727465278=86, srv280976307=61, srv1857306225=42, srv1010996446=0, srv1912110496=47, srv1919827381=48, srv761262433=88, srv333257660=68, srv1907254375=45, srv1286650195=15, srv1244425057=14, srv1458090525=26, srv980211303=99, srv1709530603=37, srv1613618982=29, srv1929332035=49, srv671361280=83, srv1908437452=46, srv1084181127=2, srv2058447962=53, srv1165094100=9, srv1879951075=44, srv460146061=74, srv830548871=90, srv1871253989=43, srv653161803=82, srv1106969418=6, srv1152671314=8, srv351514047=69, srv919822873=95, srv964706086=98, srv1177374108=10, srv1704090665=35, srv1431245727=25, srv551061371=77, srv1086155883=3, srv123770847=13, srv1106367621=5, srv226104022=58, srv154091362=28, srv369233892=70, srv1819006658=40, srv240363101=59, srv833017864=91, srv1969719604=50, srv1212461688=12, srv938125299=96, srv1412032973=23, srv2136583956=56, srv29975801=64, srv1098140890=4, srv132197638=18, srv492983775=75, srv2069860156=54, srv5854385=79, srv197156795=51, srv1709075086=36, srv630002184=80, srv544264465=76, srv885983827=92, srv1298350669=16, srv1642612350=32, srv1146564879=7, srv1377663047=20, srv134655009=19, srv320771678=67, srv292695006=63, srv901324713=93, srv1321661093=17, srv1680364082=33, srv1634015851=30, srv1825475353=41, srv1421239077=24, srv1189579352=11, srv729595772=87, srv2101955005=55, srv286853314=62, srv958372030=97, srv30992519=66, srv1806642825=39, srv1399861016=21, srv1464736648=27, srv913308369=94, srv652141122=81, srv1411515675=22, srv1636343897=31, srv673413388=84, srv1073316192=1, srv805050468=89, srv2044750023=52, srv224069938=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-22 10:54:04,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-22 10:54:04,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-22 10:54:04,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-22 10:54:04,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-22 10:54:04,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-22 10:54:04,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-22 10:54:04,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-22 10:54:04,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-22 10:54:04,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-22 10:54:04,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-22 10:54:04,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-22 10:54:04,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-22 10:54:04,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-22 10:54:04,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-22 10:54:04,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-22 10:54:04,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-22 10:54:04,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-22 10:54:04,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-22 10:54:04,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-22 10:54:04,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-22 10:54:04,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-22 10:54:04,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-22 10:54:04,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-22 10:54:04,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-22 10:54:04,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-22 10:54:04,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-22 10:54:04,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-22 10:54:04,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-22 10:54:04,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-22 10:54:04,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-22 10:54:04,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-22 10:54:04,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-22 10:54:04,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-22 10:54:04,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-22 10:54:04,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-22 10:54:04,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-22 10:54:04,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-22 10:54:04,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-22 10:54:04,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-22 10:54:04,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-22 10:54:04,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-22 10:54:04,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-22 10:54:04,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-22 10:54:04,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-22 10:54:04,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-22 10:54:04,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-22 10:54:04,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-22 10:54:04,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-22 10:54:04,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-22 10:54:04,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-22 10:54:04,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-22 10:54:04,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-22 10:54:04,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-22 10:54:04,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-22 10:54:04,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-22 10:54:04,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-22 10:54:04,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-22 10:54:04,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-22 10:54:04,824 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02340137585845833); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039985968358217015, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02340137585845833); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.015083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.049833333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:54:04,824 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-22 10:54:04,825 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01835825046526329, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02340137585845833); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039985968358217015, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02340137585845833); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.015083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.049833333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-22 10:54:05,055 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 331 ms to try 5000 different iterations. Found a solution that moves 1035 regions; Going from a computed imbalance of 0.01835825046526329 to a new imbalance of 8.402850308304445E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073312584215567194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010551567561374217); MoveCostFunction : (multiplier=7.0, imbalance=0.115, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0073312584215567194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:54:05,057 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1010996446:81 , srv1688602056:82 , srv1907254375:82 , srv1073316192:83 , srv1165094100:83 , srv1177374108:83 , srv123770847:83 , srv1871253989:83 , srv226104022:83 , srv30992519:83 , srv980211303:83 , srv1298350669:84 , srv1321661093:84 , srv1411515675:84 , srv1642612350:84 , srv1908437452:84 , srv1912110496:84 , srv2044750023:84 , srv2069860156:84 , srv292695006:84 , srv351514047:84 , srv5854385:84 , srv652141122:84 , srv885983827:84 , srv901324713:84 , srv938125299:84 , srv1106969418:85 , srv1152671314:85 , srv134655009:85 , srv1399861016:85 , srv1412032973:85 , srv1421239077:85 , srv1680364082:85 , srv1709530603:85 , srv1806642825:85 , srv1819006658:85 , srv1857306225:85 , srv1929332035:85 , srv197156795:85 , srv2058447962:85 , srv240363101:85 , srv280976307:85 , srv29975801:85 , srv320771678:85 , srv369233892:85 , srv381842997:85 , srv392489075:85 , srv551061371:85 , srv653161803:85 , srv805050468:85 , srv1098140890:86 , srv1189579352:86 , srv1244425057:86 , srv1286650195:86 , srv132197638:86 , srv1377663047:86 , srv1431245727:86 , srv1704090665:86 , srv1753623144:86 , srv1879951075:86 , srv1919827381:86 , srv333257660:86 , srv373058627:86 , srv460146061:86 , srv544264465:86 , srv560876083:86 , srv630002184:86 , srv671361280:86 , srv673413388:86 , srv68709965:86 , srv729595772:86 , srv761262433:86 , srv830548871:86 , srv919822873:86 , srv1086155883:87 , srv1212461688:87 , srv1458090525:87 , srv1464736648:87 , srv1613618982:87 , srv1634015851:87 , srv1636343897:87 , srv1709075086:87 , srv2101955005:87 , srv2136583956:87 , srv224069938:87 , srv246339268:87 , srv833017864:87 , srv913308369:87 , srv964706086:87 , srv1106367621:88 , srv1146564879:88 , srv154091362:88 , srv1825475353:88 , srv286853314:88 , srv30763917:88 , srv492983775:88 , srv727465278:88 , srv958372030:88 , srv1084181127:89 , srv1969719604:541 } 2023-02-22 10:54:05,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-22 10:54:05,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv68709965=85, srv246339268=60, srv560876083=78, srv392489075=73, srv1753623144=38, srv373058627=71, srv381842997=72, srv30763917=65, srv1688602056=34, srv727465278=86, srv280976307=61, srv1857306225=42, srv1010996446=0, srv1912110496=47, srv1919827381=48, srv761262433=88, srv333257660=68, srv1907254375=45, srv1286650195=15, srv1244425057=14, srv1458090525=26, srv980211303=99, srv1709530603=37, srv1613618982=29, srv1929332035=49, srv671361280=83, srv1908437452=46, srv1084181127=2, srv2058447962=53, srv1165094100=9, srv1879951075=44, srv460146061=74, srv830548871=90, srv1871253989=43, srv653161803=82, srv1106969418=6, srv1152671314=8, srv351514047=69, srv919822873=95, srv964706086=98, srv1177374108=10, srv1704090665=35, srv1431245727=25, srv551061371=77, srv1086155883=3, srv123770847=13, srv1106367621=5, srv226104022=58, srv154091362=28, srv369233892=70, srv1819006658=40, srv240363101=59, srv833017864=91, srv1969719604=50, srv1212461688=12, srv938125299=96, srv1412032973=23, srv2136583956=56, srv29975801=64, srv1098140890=4, srv132197638=18, srv492983775=75, srv2069860156=54, srv5854385=79, srv197156795=51, srv1709075086=36, srv630002184=80, srv544264465=76, srv885983827=92, srv1298350669=16, srv1642612350=32, srv1146564879=7, srv1377663047=20, srv134655009=19, srv320771678=67, srv292695006=63, srv901324713=93, srv1321661093=17, srv1680364082=33, srv1634015851=30, srv1825475353=41, srv1421239077=24, srv1189579352=11, srv729595772=87, srv2101955005=55, srv286853314=62, srv958372030=97, srv30992519=66, srv1806642825=39, srv1399861016=21, srv1464736648=27, srv913308369=94, srv652141122=81, srv1411515675=22, srv1636343897=31, srv673413388=84, srv1073316192=1, srv805050468=89, srv2044750023=52, srv224069938=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-22 10:54:05,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-22 10:54:05,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-22 10:54:05,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-22 10:54:05,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-22 10:54:05,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-22 10:54:05,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-22 10:54:05,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-22 10:54:05,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-22 10:54:05,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-22 10:54:05,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-22 10:54:05,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-22 10:54:05,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-22 10:54:05,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-22 10:54:05,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-22 10:54:05,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-22 10:54:05,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-22 10:54:05,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-22 10:54:05,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-22 10:54:05,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-22 10:54:05,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-22 10:54:05,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-22 10:54:05,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-22 10:54:05,192 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073312584215567194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010551567561374217); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0073312584215567194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:54:05,192 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-22 10:54:05,193 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.330378059324007E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0073312584215567194); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010551567561374217); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0073312584215567194); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-22 10:54:05,567 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 480 ms to try 5000 different iterations. Found a solution that moves 265 regions; Going from a computed imbalance of 8.330378059324007E-4 to a new imbalance of 3.7966904213965654E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024618298195866546); MoveCostFunction : (multiplier=7.0, imbalance=0.029444444444444443, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:54:05,568 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1106367621:82 , srv1634015851:83 , srv1709075086:83 , srv1084181127:84 , srv1098140890:84 , srv1146564879:84 , srv1177374108:84 , srv1212461688:84 , srv123770847:84 , srv1298350669:84 , srv132197638:84 , srv1431245727:84 , srv154091362:84 , srv1680364082:84 , srv1709530603:84 , srv1819006658:84 , srv1857306225:84 , srv1871253989:84 , srv1908437452:84 , srv1929332035:84 , srv197156795:84 , srv224069938:84 , srv280976307:84 , srv286853314:84 , srv30763917:84 , srv369233892:84 , srv381842997:84 , srv392489075:84 , srv492983775:84 , srv560876083:84 , srv671361280:84 , srv68709965:84 , srv727465278:84 , srv729595772:84 , srv761262433:84 , srv830548871:84 , srv833017864:84 , srv958372030:84 , srv964706086:84 , srv1010996446:85 , srv1073316192:85 , srv1086155883:85 , srv1106969418:85 , srv1165094100:85 , srv1189579352:85 , srv1244425057:85 , srv1286650195:85 , srv1321661093:85 , srv134655009:85 , srv1377663047:85 , srv1399861016:85 , srv1411515675:85 , srv1412032973:85 , srv1421239077:85 , srv1458090525:85 , srv1613618982:85 , srv1642612350:85 , srv1704090665:85 , srv1753623144:85 , srv1806642825:85 , srv1879951075:85 , srv1907254375:85 , srv1912110496:85 , srv1919827381:85 , srv2058447962:85 , srv2101955005:85 , srv226104022:85 , srv240363101:85 , srv246339268:85 , srv292695006:85 , srv29975801:85 , srv320771678:85 , srv333257660:85 , srv351514047:85 , srv373058627:85 , srv460146061:85 , srv544264465:85 , srv551061371:85 , srv5854385:85 , srv630002184:85 , srv652141122:85 , srv653161803:85 , srv673413388:85 , srv805050468:85 , srv885983827:85 , srv901324713:85 , srv913308369:85 , srv919822873:85 , srv938125299:85 , srv980211303:85 , srv1152671314:86 , srv1464736648:86 , srv1636343897:86 , srv1688602056:86 , srv1825475353:86 , srv2044750023:86 , srv2069860156:86 , srv2136583956:86 , srv30992519:86 , srv1969719604:619 } 2023-02-22 10:54:05,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-22 10:54:05,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv68709965=85, srv246339268=60, srv560876083=78, srv392489075=73, srv1753623144=38, srv373058627=71, srv381842997=72, srv30763917=65, srv1688602056=34, srv727465278=86, srv280976307=61, srv1857306225=42, srv1010996446=0, srv1912110496=47, srv1919827381=48, srv761262433=88, srv333257660=68, srv1907254375=45, srv1286650195=15, srv1244425057=14, srv1458090525=26, srv980211303=99, srv1709530603=37, srv1613618982=29, srv1929332035=49, srv671361280=83, srv1908437452=46, srv1084181127=2, srv2058447962=53, srv1165094100=9, srv1879951075=44, srv460146061=74, srv830548871=90, srv1871253989=43, srv653161803=82, srv1106969418=6, srv1152671314=8, srv351514047=69, srv919822873=95, srv964706086=98, srv1177374108=10, srv1704090665=35, srv1431245727=25, srv551061371=77, srv1086155883=3, srv123770847=13, srv1106367621=5, srv226104022=58, srv154091362=28, srv369233892=70, srv1819006658=40, srv240363101=59, srv833017864=91, srv1969719604=50, srv1212461688=12, srv938125299=96, srv1412032973=23, srv2136583956=56, srv29975801=64, srv1098140890=4, srv132197638=18, srv492983775=75, srv2069860156=54, srv5854385=79, srv197156795=51, srv1709075086=36, srv630002184=80, srv544264465=76, srv885983827=92, srv1298350669=16, srv1642612350=32, srv1146564879=7, srv1377663047=20, srv134655009=19, srv320771678=67, srv292695006=63, srv901324713=93, srv1321661093=17, srv1680364082=33, srv1634015851=30, srv1825475353=41, srv1421239077=24, srv1189579352=11, srv729595772=87, srv2101955005=55, srv286853314=62, srv958372030=97, srv30992519=66, srv1806642825=39, srv1399861016=21, srv1464736648=27, srv913308369=94, srv652141122=81, srv1411515675=22, srv1636343897=31, srv673413388=84, srv1073316192=1, srv805050468=89, srv2044750023=52, srv224069938=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-22 10:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-22 10:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-22 10:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-22 10:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-22 10:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-22 10:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-22 10:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-22 10:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-22 10:54:05,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-22 10:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-22 10:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-22 10:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-22 10:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-22 10:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-22 10:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-22 10:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-22 10:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-22 10:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-22 10:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-22 10:54:05,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-22 10:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-22 10:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-22 10:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-22 10:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-22 10:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-22 10:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-22 10:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-22 10:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-22 10:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-22 10:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-22 10:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-22 10:54:05,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-22 10:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-22 10:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-22 10:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-22 10:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-22 10:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-22 10:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-22 10:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-22 10:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-22 10:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-22 10:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-22 10:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-22 10:54:05,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-22 10:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-22 10:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-22 10:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-22 10:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-22 10:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-22 10:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-22 10:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-22 10:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-22 10:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-22 10:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-22 10:54:05,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-22 10:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-22 10:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-22 10:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-22 10:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-22 10:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-22 10:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-22 10:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-22 10:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-22 10:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-22 10:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-22 10:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-22 10:54:05,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-22 10:54:05,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-22 10:54:05,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-22 10:54:05,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-22 10:54:05,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-22 10:54:05,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-22 10:54:05,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-22 10:54:05,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-22 10:54:05,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-22 10:54:05,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-22 10:54:05,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-22 10:54:05,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-22 10:54:05,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-22 10:54:05,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-22 10:54:05,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-22 10:54:05,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-22 10:54:05,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-22 10:54:05,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-22 10:54:05,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-22 10:54:05,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-22 10:54:05,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-22 10:54:05,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-22 10:54:05,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-22 10:54:05,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-22 10:54:05,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-22 10:54:05,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-22 10:54:05,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-22 10:54:05,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-22 10:54:05,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-22 10:54:05,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-22 10:54:05,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-22 10:54:05,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-22 10:54:05,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-22 10:54:05,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-22 10:54:05,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-22 10:54:05,712 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024618298195866546); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:54:05,712 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-22 10:54:05,713 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.611133455408007E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024618298195866546); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-22 10:54:05,971 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 361 ms to try 5000 different iterations. Found a solution that moves 49 regions; Going from a computed imbalance of 3.611133455408007E-5 to a new imbalance of 6.741091005607033E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0054444444444444445); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:54:05,971 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1464736648:83 , srv1636343897:83 , srv1806642825:83 , srv1825475353:83 , srv1919827381:83 , srv2069860156:83 , srv29975801:83 , srv351514047:83 , srv673413388:83 , srv885983827:83 , srv938125299:83 , srv1010996446:84 , srv1073316192:84 , srv1084181127:84 , srv1086155883:84 , srv1098140890:84 , srv1106367621:84 , srv1106969418:84 , srv1146564879:84 , srv1152671314:84 , srv1165094100:84 , srv1189579352:84 , srv1212461688:84 , srv1286650195:84 , srv1298350669:84 , srv1321661093:84 , srv134655009:84 , srv1377663047:84 , srv1399861016:84 , srv1411515675:84 , srv1412032973:84 , srv1421239077:84 , srv1431245727:84 , srv1458090525:84 , srv154091362:84 , srv1613618982:84 , srv1634015851:84 , srv1642612350:84 , srv1680364082:84 , srv1688602056:84 , srv1704090665:84 , srv1753623144:84 , srv1819006658:84 , srv1871253989:84 , srv1879951075:84 , srv1907254375:84 , srv1912110496:84 , srv1929332035:84 , srv197156795:84 , srv2044750023:84 , srv2058447962:84 , srv2101955005:84 , srv2136583956:84 , srv224069938:84 , srv226104022:84 , srv240363101:84 , srv246339268:84 , srv280976307:84 , srv286853314:84 , srv292695006:84 , srv30763917:84 , srv30992519:84 , srv320771678:84 , srv333257660:84 , srv373058627:84 , srv381842997:84 , srv392489075:84 , srv460146061:84 , srv492983775:84 , srv544264465:84 , srv551061371:84 , srv560876083:84 , srv5854385:84 , srv630002184:84 , srv652141122:84 , srv653161803:84 , srv671361280:84 , srv68709965:84 , srv729595772:84 , srv761262433:84 , srv805050468:84 , srv833017864:84 , srv901324713:84 , srv913308369:84 , srv919822873:84 , srv958372030:84 , srv964706086:84 , srv980211303:84 , srv1177374108:85 , srv123770847:85 , srv1244425057:85 , srv132197638:85 , srv1709075086:85 , srv1709530603:85 , srv1857306225:85 , srv1908437452:85 , srv369233892:85 , srv727465278:85 , srv830548871:85 , srv1969719604:684 } 2023-02-22 10:54:05,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-22 10:54:05,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv68709965=85, srv246339268=60, srv560876083=78, srv392489075=73, srv1753623144=38, srv373058627=71, srv381842997=72, srv30763917=65, srv1688602056=34, srv727465278=86, srv280976307=61, srv1857306225=42, srv1010996446=0, srv1912110496=47, srv1919827381=48, srv761262433=88, srv333257660=68, srv1907254375=45, srv1286650195=15, srv1244425057=14, srv1458090525=26, srv980211303=99, srv1709530603=37, srv1613618982=29, srv1929332035=49, srv671361280=83, srv1908437452=46, srv1084181127=2, srv2058447962=53, srv1165094100=9, srv1879951075=44, srv460146061=74, srv830548871=90, srv1871253989=43, srv653161803=82, srv1106969418=6, srv1152671314=8, srv351514047=69, srv919822873=95, srv964706086=98, srv1177374108=10, srv1704090665=35, srv1431245727=25, srv551061371=77, srv1086155883=3, srv123770847=13, srv1106367621=5, srv226104022=58, srv154091362=28, srv369233892=70, srv1819006658=40, srv240363101=59, srv833017864=91, srv1969719604=50, srv1212461688=12, srv938125299=96, srv1412032973=23, srv2136583956=56, srv29975801=64, srv1098140890=4, srv132197638=18, srv492983775=75, srv2069860156=54, srv5854385=79, srv197156795=51, srv1709075086=36, srv630002184=80, srv544264465=76, srv885983827=92, srv1298350669=16, srv1642612350=32, srv1146564879=7, srv1377663047=20, srv134655009=19, srv320771678=67, srv292695006=63, srv901324713=93, srv1321661093=17, srv1680364082=33, srv1634015851=30, srv1825475353=41, srv1421239077=24, srv1189579352=11, srv729595772=87, srv2101955005=55, srv286853314=62, srv958372030=97, srv30992519=66, srv1806642825=39, srv1399861016=21, srv1464736648=27, srv913308369=94, srv652141122=81, srv1411515675=22, srv1636343897=31, srv673413388=84, srv1073316192=1, srv805050468=89, srv2044750023=52, srv224069938=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-22 10:54:06,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-22 10:54:06,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-22 10:54:06,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-22 10:54:06,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-22 10:54:06,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-22 10:54:06,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-22 10:54:06,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-22 10:54:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-22 10:54:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-22 10:54:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-22 10:54:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-22 10:54:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-22 10:54:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-22 10:54:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-22 10:54:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-22 10:54:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-22 10:54:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-22 10:54:06,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-22 10:54:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-22 10:54:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-22 10:54:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-22 10:54:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-22 10:54:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-22 10:54:06,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-22 10:54:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-22 10:54:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-22 10:54:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-22 10:54:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-22 10:54:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-22 10:54:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-22 10:54:06,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-22 10:54:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-22 10:54:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-22 10:54:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-22 10:54:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-22 10:54:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-22 10:54:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-22 10:54:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-22 10:54:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-22 10:54:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-22 10:54:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-22 10:54:06,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-22 10:54:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-22 10:54:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-22 10:54:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-22 10:54:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-22 10:54:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-22 10:54:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-22 10:54:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-22 10:54:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-22 10:54:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-22 10:54:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-22 10:54:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-22 10:54:06,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-22 10:54:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-22 10:54:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-22 10:54:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-22 10:54:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-22 10:54:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-22 10:54:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-22 10:54:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-22 10:54:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-22 10:54:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-22 10:54:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-22 10:54:06,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-22 10:54:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-22 10:54:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-22 10:54:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-22 10:54:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-22 10:54:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-22 10:54:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-22 10:54:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-22 10:54:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-22 10:54:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-22 10:54:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-22 10:54:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-22 10:54:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-22 10:54:06,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-22 10:54:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-22 10:54:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-22 10:54:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-22 10:54:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-22 10:54:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-22 10:54:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-22 10:54:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-22 10:54:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-22 10:54:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-22 10:54:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-22 10:54:06,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-22 10:54:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-22 10:54:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-22 10:54:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-22 10:54:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-22 10:54:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-22 10:54:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-22 10:54:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-22 10:54:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-22 10:54:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-22 10:54:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-22 10:54:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-22 10:54:06,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-22 10:54:06,131 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:54:06,131 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1464736648:83 , srv1636343897:83 , srv1806642825:83 , srv1825475353:83 , srv1919827381:83 , srv2069860156:83 , srv29975801:83 , srv351514047:83 , srv673413388:83 , srv885983827:83 , srv938125299:83 , srv1010996446:84 , srv1073316192:84 , srv1084181127:84 , srv1086155883:84 , srv1098140890:84 , srv1106367621:84 , srv1106969418:84 , srv1146564879:84 , srv1152671314:84 , srv1165094100:84 , srv1189579352:84 , srv1212461688:84 , srv1286650195:84 , srv1298350669:84 , srv1321661093:84 , srv134655009:84 , srv1377663047:84 , srv1399861016:84 , srv1411515675:84 , srv1412032973:84 , srv1421239077:84 , srv1431245727:84 , srv1458090525:84 , srv154091362:84 , srv1613618982:84 , srv1634015851:84 , srv1642612350:84 , srv1680364082:84 , srv1688602056:84 , srv1704090665:84 , srv1753623144:84 , srv1819006658:84 , srv1871253989:84 , srv1879951075:84 , srv1907254375:84 , srv1912110496:84 , srv1929332035:84 , srv197156795:84 , srv2044750023:84 , srv2058447962:84 , srv2101955005:84 , srv2136583956:84 , srv224069938:84 , srv226104022:84 , srv240363101:84 , srv246339268:84 , srv280976307:84 , srv286853314:84 , srv292695006:84 , srv30763917:84 , srv30992519:84 , srv320771678:84 , srv333257660:84 , srv373058627:84 , srv381842997:84 , srv392489075:84 , srv460146061:84 , srv492983775:84 , srv544264465:84 , srv551061371:84 , srv560876083:84 , srv5854385:84 , srv630002184:84 , srv652141122:84 , srv653161803:84 , srv671361280:84 , srv68709965:84 , srv729595772:84 , srv761262433:84 , srv805050468:84 , srv833017864:84 , srv901324713:84 , srv913308369:84 , srv919822873:84 , srv958372030:84 , srv964706086:84 , srv980211303:84 , srv1177374108:85 , srv123770847:85 , srv1244425057:85 , srv132197638:85 , srv1709075086:85 , srv1709530603:85 , srv1857306225:85 , srv1908437452:85 , srv369233892:85 , srv727465278:85 , srv830548871:85 , srv1969719604:684 } 2023-02-22 10:54:06,171 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=330 (was 272) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=7699 (was 8308) 2023-02-22 10:54:06,186 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=330, ProcessCount=172, AvailableMemoryMB=7699 2023-02-22 10:54:06,187 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-22 10:54:06,188 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-02-22 10:54:06,203 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1090480828:84 , srv1107363641:84 , srv1118196110:84 , srv1186038875:84 , srv1202599674:84 , srv1378463451:84 , srv1396466894:84 , srv1538995399:84 , srv1558228749:84 , srv1564357209:84 , srv1566067712:84 , srv172547980:84 , srv1754330797:84 , srv1759509538:84 , srv1970374801:84 , srv2071324341:84 , srv211026522:84 , srv31728308:84 , srv397420921:84 , srv463639114:84 , srv611072940:84 , srv622016463:84 , srv649368797:84 , srv702027669:84 , srv833406578:84 , srv879188102:84 , srv937946468:84 , srv977247794:84 , srv983797451:84 , srv831800067:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-02-22 10:54:06,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-22 10:54:06,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1396466894=6, srv879188102=26, srv463639114=19, srv1090480828=0, srv1566067712=10, srv1558228749=8, srv1538995399=7, srv983797451=29, srv2071324341=15, srv397420921=18, srv622016463=21, srv1202599674=4, srv831800067=24, srv1118196110=2, srv1186038875=3, srv702027669=23, srv977247794=28, srv1378463451=5, srv1759509538=13, srv611072940=20, srv937946468=27, srv31728308=17, srv649368797=22, srv1107363641=1, srv1754330797=12, srv1564357209=9, srv211026522=16, srv833406578=25, srv172547980=11, srv1970374801=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-22 10:54:06,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-22 10:54:06,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-22 10:54:06,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-22 10:54:06,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-22 10:54:06,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-22 10:54:06,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-22 10:54:06,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-22 10:54:06,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-22 10:54:06,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-22 10:54:06,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-22 10:54:06,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-22 10:54:06,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-22 10:54:06,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-22 10:54:06,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-22 10:54:06,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-22 10:54:06,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-22 10:54:06,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-22 10:54:06,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-22 10:54:06,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-22 10:54:06,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-22 10:54:06,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-22 10:54:06,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-22 10:54:06,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-22 10:54:06,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-22 10:54:06,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-22 10:54:06,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-22 10:54:06,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-22 10:54:06,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-22 10:54:06,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-22 10:54:06,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-22 10:54:06,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-22 10:54:06,251 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1423624636440917, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:54:06,252 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912641622738659, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1423624636440917, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-02-22 10:56:06,228 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6713381 different iterations. Found a solution that moves 1852 regions; Going from a computed imbalance of 0.9912641622738659 to a new imbalance of 4.322660390073086E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6859259259259259, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:56:06,235 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1090480828:90 , srv1107363641:90 , srv1118196110:90 , srv1186038875:90 , srv1202599674:90 , srv1378463451:90 , srv1396466894:90 , srv1538995399:90 , srv1558228749:90 , srv1564357209:90 , srv1566067712:90 , srv172547980:90 , srv1754330797:90 , srv1759509538:90 , srv1970374801:90 , srv2071324341:90 , srv211026522:90 , srv31728308:90 , srv397420921:90 , srv463639114:90 , srv611072940:90 , srv622016463:90 , srv649368797:90 , srv702027669:90 , srv831800067:90 , srv833406578:90 , srv879188102:90 , srv937946468:90 , srv977247794:90 , srv983797451:90 } 2023-02-22 10:56:06,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-22 10:56:06,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1396466894=6, srv879188102=26, srv463639114=19, srv1090480828=0, srv1566067712=10, srv1558228749=8, srv1538995399=7, srv983797451=29, srv2071324341=15, srv397420921=18, srv622016463=21, srv1202599674=4, srv831800067=24, srv1118196110=2, srv1186038875=3, srv702027669=23, srv977247794=28, srv1378463451=5, srv1759509538=13, srv611072940=20, srv937946468=27, srv31728308=17, srv649368797=22, srv1107363641=1, srv1754330797=12, srv1564357209=9, srv211026522=16, srv833406578=25, srv172547980=11, srv1970374801=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-22 10:56:06,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-22 10:56:06,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-22 10:56:06,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-22 10:56:06,259 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-22 10:56:06,259 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1090480828:90 , srv1107363641:90 , srv1118196110:90 , srv1186038875:90 , srv1202599674:90 , srv1378463451:90 , srv1396466894:90 , srv1538995399:90 , srv1558228749:90 , srv1564357209:90 , srv1566067712:90 , srv172547980:90 , srv1754330797:90 , srv1759509538:90 , srv1970374801:90 , srv2071324341:90 , srv211026522:90 , srv31728308:90 , srv397420921:90 , srv463639114:90 , srv611072940:90 , srv622016463:90 , srv649368797:90 , srv702027669:90 , srv831800067:90 , srv833406578:90 , srv879188102:90 , srv937946468:90 , srv977247794:90 , srv983797451:90 } 2023-02-22 10:56:06,271 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=181 (was 330), ProcessCount=172 (was 172), AvailableMemoryMB=7583 (was 7699)