2024-02-11 22:53:10,319 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-02-11 22:53:10,626 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-02-11 22:53:10,730 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-02-11 22:53:10,794 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-11 22:53:10,801 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-11 22:53:10,813 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=271, ProcessCount=170, AvailableMemoryMB=3128 2024-02-11 22:53:10,815 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-11 22:53:10,816 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-11 22:53:10,932 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010679784:84 , srv1011278569:84 , srv1018204128:84 , srv1034434036:84 , srv1059272538:84 , srv1092321123:84 , srv1104560067:84 , srv1110969436:84 , srv1118743770:84 , srv1129679112:84 , srv1161416438:84 , srv1163317944:84 , srv1251876975:84 , srv128174861:84 , srv1292692187:84 , srv1316272202:84 , srv1323619189:84 , srv1355890453:84 , srv138849533:84 , srv1406665682:84 , srv1438481623:84 , srv146092503:84 , srv1464790378:84 , srv1483045045:84 , srv1514039906:84 , srv1517818692:84 , srv1522299785:84 , srv1539365441:84 , srv1550644895:84 , srv1588322877:84 , srv162300567:84 , srv1632175473:84 , srv1633591089:84 , srv1653227675:84 , srv1656487593:84 , srv166278936:84 , srv1668213542:84 , srv1669835438:84 , srv1675759208:84 , srv1740365315:84 , srv1747142862:84 , srv1759664461:84 , srv1770940651:84 , srv1778162838:84 , srv1794729381:84 , srv1799238939:84 , srv1830189018:84 , srv1887813980:84 , srv1915300983:84 , srv1915301776:84 , srv192478383:84 , srv1926658658:84 , srv1937217598:84 , srv1948570661:84 , srv1989209601:84 , srv2008655726:84 , srv2016913308:84 , srv210369158:84 , srv2116673056:84 , srv2125384353:84 , srv283659534:84 , srv324548761:84 , srv358568577:84 , srv386397646:84 , srv401196382:84 , srv405698763:84 , srv413185037:84 , srv419471270:84 , srv491810409:84 , srv491821698:84 , srv494499015:84 , srv504963305:84 , srv519989320:84 , srv523003070:84 , srv577504467:84 , srv589160946:84 , srv598690286:84 , srv607872029:84 , srv648730834:84 , srv657350567:84 , srv674052822:84 , srv710053092:84 , srv712731374:84 , srv717109190:84 , srv767984058:84 , srv774708275:84 , srv79076126:84 , srv790923500:84 , srv82687239:84 , srv860786868:84 , srv871142092:84 , srv873641822:84 , srv88427127:84 , srv918469565:84 , srv931702835:84 , srv932504779:84 , srv960677666:84 , srv981868871:84 , srv986395939:84 , srv380088699:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-02-11 22:53:10,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 22:53:10,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv405698763=66, srv1887813980=47, srv2125384353=59, srv1118743770=8, srv1915300983=48, srv2116673056=58, srv380088699=63, srv1830189018=46, srv2016913308=56, srv774708275=86, srv648730834=79, srv1483045045=23, srv607872029=78, srv146092503=21, srv88427127=93, srv657350567=80, srv1632175473=31, srv1163317944=11, srv1747142862=40, srv1059272538=4, srv1464790378=22, srv491821698=70, srv1018204128=2, srv1355890453=17, srv674052822=81, srv1104560067=6, srv1292692187=14, srv192478383=50, srv82687239=89, srv494499015=71, srv1653227675=33, srv767984058=85, srv491810409=69, srv598690286=77, srv1539365441=27, srv1011278569=1, srv1034434036=3, srv1926658658=51, srv1668213542=36, srv918469565=94, srv1550644895=28, srv138849533=18, srv2008655726=55, srv401196382=65, srv1251876975=12, srv981868871=98, srv1633591089=32, srv1669835438=37, srv717109190=84, srv413185037=67, srv871142092=91, srv1948570661=53, srv589160946=76, srv386397646=64, srv1517818692=25, srv358568577=62, srv419471270=68, srv1799238939=45, srv210369158=57, srv1406665682=19, srv1514039906=24, srv873641822=92, srv324548761=61, srv1675759208=38, srv523003070=74, srv283659534=60, srv1110969436=7, srv79076126=87, srv932504779=96, srv931702835=95, srv1656487593=34, srv1010679784=0, srv1759664461=41, srv986395939=99, srv519989320=73, srv1161416438=10, srv960677666=97, srv1092321123=5, srv1438481623=20, srv1770940651=42, srv1915301776=49, srv1129679112=9, srv166278936=35, srv710053092=82, srv712731374=83, srv1740365315=39, srv1778162838=43, srv504963305=72, srv1937217598=52, srv1522299785=26, srv1794729381=44, srv128174861=13, srv1316272202=15, srv162300567=30, srv860786868=90, srv790923500=88, srv1323619189=16, srv577504467=75, srv1588322877=29, srv1989209601=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 22:53:11,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 22:53:11,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 22:53:11,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 22:53:11,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 22:53:11,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 22:53:11,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 22:53:11,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 22:53:11,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 22:53:11,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 22:53:11,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 22:53:11,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 22:53:11,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 22:53:11,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 22:53:11,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 22:53:11,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 22:53:11,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 22:53:11,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 22:53:11,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 22:53:11,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 22:53:11,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 22:53:11,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 22:53:11,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 22:53:11,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 22:53:11,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 22:53:11,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 22:53:11,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 22:53:11,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 22:53:11,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 22:53:11,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 22:53:11,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 22:53:11,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 22:53:11,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 22:53:11,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 22:53:11,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 22:53:11,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 22:53:11,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 22:53:11,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 22:53:11,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 22:53:11,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 22:53:11,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 22:53:11,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 22:53:11,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 22:53:11,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 22:53:11,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 22:53:11,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 22:53:11,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 22:53:11,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 22:53:11,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 22:53:11,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 22:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 22:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 22:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 22:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 22:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 22:53:11,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 22:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 22:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 22:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 22:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 22:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 22:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 22:53:11,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 22:53:11,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 22:53:11,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 22:53:11,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 22:53:11,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 22:53:11,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 22:53:11,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 22:53:11,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 22:53:11,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 22:53:11,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 22:53:11,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 22:53:11,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 22:53:11,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 22:53:11,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 22:53:11,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 22:53:11,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 22:53:11,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 22:53:11,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 22:53:11,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 22:53:11,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 22:53:11,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 22:53:11,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 22:53:11,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 22:53:11,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 22:53:11,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 22:53:11,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 22:53:11,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 22:53:11,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 22:53:11,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 22:53:11,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 22:53:11,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 22:53:11,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 22:53:11,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 22:53:11,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 22:53:11,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 22:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 22:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 22:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 22:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 22:53:11,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 22:53:11,255 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:53:11,255 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 22:53:11,255 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 22:53:11,723 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 645 ms to try 5000 different iterations. Found a solution that moves 4663 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18302666482812624. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038598725092293756, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054884840156570976, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5181111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038598725092293756, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17858333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:53:11,735 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1355890453:83 , srv1438481623:83 , srv1483045045:83 , srv1794729381:83 , srv192478383:83 , srv401196382:83 , srv491821698:83 , srv774708275:83 , srv1018204128:84 , srv1251876975:84 , srv1323619189:84 , srv1406665682:84 , srv1522299785:84 , srv1550644895:84 , srv1632175473:84 , srv1668213542:84 , srv1669835438:84 , srv1740365315:84 , srv1770940651:84 , srv324548761:84 , srv491810409:84 , srv523003070:84 , srv648730834:84 , srv717109190:84 , srv918469565:84 , srv981868871:84 , srv1059272538:85 , srv1110969436:85 , srv1633591089:85 , srv1915300983:85 , srv1948570661:85 , srv494499015:85 , srv589160946:85 , srv657350567:85 , srv1161416438:86 , srv1292692187:86 , srv138849533:86 , srv162300567:86 , srv166278936:86 , srv1675759208:86 , srv1747142862:86 , srv1915301776:86 , srv2008655726:86 , srv210369158:86 , srv358568577:86 , srv413185037:86 , srv504963305:86 , srv79076126:86 , srv88427127:86 , srv931702835:86 , srv986395939:86 , srv1011278569:87 , srv1034434036:87 , srv1092321123:87 , srv1118743770:87 , srv1129679112:87 , srv1316272202:87 , srv1514039906:87 , srv1517818692:87 , srv1539365441:87 , srv1588322877:87 , srv1653227675:87 , srv1799238939:87 , srv1830189018:87 , srv1926658658:87 , srv1937217598:87 , srv1989209601:87 , srv2125384353:87 , srv283659534:87 , srv386397646:87 , srv419471270:87 , srv598690286:87 , srv607872029:87 , srv710053092:87 , srv1104560067:88 , srv1759664461:88 , srv2016913308:88 , srv2116673056:88 , srv82687239:88 , srv960677666:88 , srv1163317944:89 , srv146092503:89 , srv1464790378:89 , srv1887813980:89 , srv405698763:89 , srv674052822:89 , srv790923500:89 , srv860786868:89 , srv932504779:89 , srv128174861:90 , srv1656487593:90 , srv519989320:90 , srv871142092:90 , srv1010679784:91 , srv873641822:92 , srv1778162838:93 , srv577504467:94 , srv712731374:94 , srv767984058:95 , srv380088699:433 } 2024-02-11 22:53:11,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 22:53:11,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv405698763=66, srv1887813980=47, srv2125384353=59, srv1118743770=8, srv1915300983=48, srv2116673056=58, srv380088699=63, srv1830189018=46, srv2016913308=56, srv774708275=86, srv648730834=79, srv1483045045=23, srv607872029=78, srv146092503=21, srv88427127=93, srv657350567=80, srv1632175473=31, srv1163317944=11, srv1747142862=40, srv1059272538=4, srv1464790378=22, srv491821698=70, srv1018204128=2, srv1355890453=17, srv674052822=81, srv1104560067=6, srv1292692187=14, srv192478383=50, srv82687239=89, srv494499015=71, srv1653227675=33, srv767984058=85, srv491810409=69, srv598690286=77, srv1539365441=27, srv1011278569=1, srv1034434036=3, srv1926658658=51, srv1668213542=36, srv918469565=94, srv1550644895=28, srv138849533=18, srv2008655726=55, srv401196382=65, srv1251876975=12, srv981868871=98, srv1633591089=32, srv1669835438=37, srv717109190=84, srv413185037=67, srv871142092=91, srv1948570661=53, srv589160946=76, srv386397646=64, srv1517818692=25, srv358568577=62, srv419471270=68, srv1799238939=45, srv210369158=57, srv1406665682=19, srv1514039906=24, srv873641822=92, srv324548761=61, srv1675759208=38, srv523003070=74, srv283659534=60, srv1110969436=7, srv79076126=87, srv932504779=96, srv931702835=95, srv1656487593=34, srv1010679784=0, srv1759664461=41, srv986395939=99, srv519989320=73, srv1161416438=10, srv960677666=97, srv1092321123=5, srv1438481623=20, srv1770940651=42, srv1915301776=49, srv1129679112=9, srv166278936=35, srv710053092=82, srv712731374=83, srv1740365315=39, srv1778162838=43, srv504963305=72, srv1937217598=52, srv1522299785=26, srv1794729381=44, srv128174861=13, srv1316272202=15, srv162300567=30, srv860786868=90, srv790923500=88, srv1323619189=16, srv577504467=75, srv1588322877=29, srv1989209601=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 22:53:11,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 22:53:11,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 22:53:11,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 22:53:11,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 22:53:11,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 22:53:11,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 22:53:11,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 22:53:11,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 22:53:11,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 22:53:11,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 22:53:11,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 22:53:11,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 22:53:11,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 22:53:11,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 22:53:11,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 22:53:11,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 22:53:11,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 22:53:11,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 22:53:11,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 22:53:11,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 22:53:11,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 22:53:11,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 22:53:11,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 22:53:11,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 22:53:11,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 22:53:11,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 22:53:11,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 22:53:11,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 22:53:11,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 22:53:11,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 22:53:11,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 22:53:11,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 22:53:11,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 22:53:11,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 22:53:11,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 22:53:11,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 22:53:11,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 22:53:11,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 22:53:11,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 22:53:11,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 22:53:11,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 22:53:11,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 22:53:11,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 22:53:11,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 22:53:11,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 22:53:11,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 22:53:11,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 22:53:11,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 22:53:11,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 22:53:11,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 22:53:11,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 22:53:11,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 22:53:11,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 22:53:11,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 22:53:11,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 22:53:11,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 22:53:11,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 22:53:11,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 22:53:11,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 22:53:11,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 22:53:11,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 22:53:11,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 22:53:11,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 22:53:11,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 22:53:11,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 22:53:11,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 22:53:11,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 22:53:11,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 22:53:11,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 22:53:11,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 22:53:11,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 22:53:11,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 22:53:11,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 22:53:11,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 22:53:11,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 22:53:11,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 22:53:11,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 22:53:11,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 22:53:11,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 22:53:11,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 22:53:11,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 22:53:11,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 22:53:11,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 22:53:11,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 22:53:11,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 22:53:11,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 22:53:11,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 22:53:11,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 22:53:11,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 22:53:11,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 22:53:11,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 22:53:11,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 22:53:11,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 22:53:11,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 22:53:11,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 22:53:11,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 22:53:11,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 22:53:11,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 22:53:11,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 22:53:11,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 22:53:11,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 22:53:11,922 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038598725092293756, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054884840156570976, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038598725092293756, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17858333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:53:11,922 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 22:53:11,922 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1829940138042619, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038598725092293756, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054884840156570976, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038598725092293756, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17858333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 22:53:12,308 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 503 ms to try 5000 different iterations. Found a solution that moves 2637 regions; Going from a computed imbalance of 0.1829940138042619 to a new imbalance of 0.01405096548244697. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021272054714097814); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03461508724343955, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.293, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021272054714097814); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:53:12,323 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv767984058:77 , srv712731374:78 , srv577504467:79 , srv128174861:81 , srv1656487593:81 , srv1778162838:81 , srv519989320:81 , srv873641822:81 , srv1118743770:82 , srv1464790378:82 , srv405698763:82 , srv932504779:82 , srv1010679784:83 , srv1514039906:83 , srv1759664461:83 , srv1887813980:83 , srv210369158:83 , srv283659534:83 , srv674052822:83 , srv790923500:83 , srv1316272202:84 , srv138849533:84 , srv146092503:84 , srv1926658658:84 , srv1937217598:84 , srv2116673056:84 , srv710053092:84 , srv79076126:84 , srv1011278569:85 , srv1034434036:85 , srv1092321123:85 , srv1129679112:85 , srv1163317944:85 , srv1323619189:85 , srv1517818692:85 , srv1799238939:85 , srv1948570661:85 , srv1989209601:85 , srv2008655726:85 , srv2016913308:85 , srv2125384353:85 , srv419471270:85 , srv589160946:85 , srv607872029:85 , srv931702835:85 , srv1104560067:86 , srv1292692187:86 , srv1539365441:86 , srv1550644895:86 , srv1588322877:86 , srv1633591089:86 , srv1669835438:86 , srv1770940651:86 , srv358568577:86 , srv413185037:86 , srv504963305:86 , srv598690286:86 , srv860786868:86 , srv871142092:86 , srv960677666:86 , srv1059272538:87 , srv162300567:87 , srv1632175473:87 , srv1653227675:87 , srv166278936:87 , srv1668213542:87 , srv1675759208:87 , srv1740365315:87 , srv1830189018:87 , srv1915301776:87 , srv386397646:87 , srv494499015:87 , srv523003070:87 , srv82687239:87 , srv88427127:87 , srv986395939:87 , srv1110969436:88 , srv1161416438:88 , srv1251876975:88 , srv1355890453:88 , srv1747142862:88 , srv1794729381:88 , srv1915300983:88 , srv401196382:88 , srv491810409:88 , srv491821698:88 , srv981868871:88 , srv1018204128:89 , srv1406665682:89 , srv1438481623:89 , srv1483045045:89 , srv192478383:89 , srv324548761:89 , srv648730834:89 , srv717109190:89 , srv774708275:89 , srv918469565:89 , srv1522299785:91 , srv657350567:91 , srv380088699:530 } 2024-02-11 22:53:12,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 22:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv405698763=66, srv1887813980=47, srv2125384353=59, srv1118743770=8, srv1915300983=48, srv2116673056=58, srv380088699=63, srv1830189018=46, srv2016913308=56, srv774708275=86, srv648730834=79, srv1483045045=23, srv607872029=78, srv146092503=21, srv88427127=93, srv657350567=80, srv1632175473=31, srv1163317944=11, srv1747142862=40, srv1059272538=4, srv1464790378=22, srv491821698=70, srv1018204128=2, srv1355890453=17, srv674052822=81, srv1104560067=6, srv1292692187=14, srv192478383=50, srv82687239=89, srv494499015=71, srv1653227675=33, srv767984058=85, srv491810409=69, srv598690286=77, srv1539365441=27, srv1011278569=1, srv1034434036=3, srv1926658658=51, srv1668213542=36, srv918469565=94, srv1550644895=28, srv138849533=18, srv2008655726=55, srv401196382=65, srv1251876975=12, srv981868871=98, srv1633591089=32, srv1669835438=37, srv717109190=84, srv413185037=67, srv871142092=91, srv1948570661=53, srv589160946=76, srv386397646=64, srv1517818692=25, srv358568577=62, srv419471270=68, srv1799238939=45, srv210369158=57, srv1406665682=19, srv1514039906=24, srv873641822=92, srv324548761=61, srv1675759208=38, srv523003070=74, srv283659534=60, srv1110969436=7, srv79076126=87, srv932504779=96, srv931702835=95, srv1656487593=34, srv1010679784=0, srv1759664461=41, srv986395939=99, srv519989320=73, srv1161416438=10, srv960677666=97, srv1092321123=5, srv1438481623=20, srv1770940651=42, srv1915301776=49, srv1129679112=9, srv166278936=35, srv710053092=82, srv712731374=83, srv1740365315=39, srv1778162838=43, srv504963305=72, srv1937217598=52, srv1522299785=26, srv1794729381=44, srv128174861=13, srv1316272202=15, srv162300567=30, srv860786868=90, srv790923500=88, srv1323619189=16, srv577504467=75, srv1588322877=29, srv1989209601=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 22:53:12,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 22:53:12,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 22:53:12,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 22:53:12,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 22:53:12,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 22:53:12,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 22:53:12,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 22:53:12,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 22:53:12,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 22:53:12,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 22:53:12,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 22:53:12,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 22:53:12,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 22:53:12,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 22:53:12,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 22:53:12,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 22:53:12,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 22:53:12,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 22:53:12,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 22:53:12,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 22:53:12,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 22:53:12,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 22:53:12,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 22:53:12,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 22:53:12,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 22:53:12,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 22:53:12,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 22:53:12,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 22:53:12,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 22:53:12,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 22:53:12,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 22:53:12,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 22:53:12,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 22:53:12,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 22:53:12,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 22:53:12,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 22:53:12,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 22:53:12,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 22:53:12,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 22:53:12,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 22:53:12,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 22:53:12,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 22:53:12,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 22:53:12,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 22:53:12,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 22:53:12,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 22:53:12,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 22:53:12,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 22:53:12,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 22:53:12,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 22:53:12,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 22:53:12,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 22:53:12,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 22:53:12,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 22:53:12,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 22:53:12,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 22:53:12,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 22:53:12,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 22:53:12,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 22:53:12,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 22:53:12,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 22:53:12,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 22:53:12,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 22:53:12,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 22:53:12,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 22:53:12,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 22:53:12,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 22:53:12,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 22:53:12,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 22:53:12,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 22:53:12,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 22:53:12,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 22:53:12,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 22:53:12,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 22:53:12,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 22:53:12,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 22:53:12,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 22:53:12,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 22:53:12,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 22:53:12,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 22:53:12,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 22:53:12,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 22:53:12,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 22:53:12,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 22:53:12,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 22:53:12,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 22:53:12,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 22:53:12,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 22:53:12,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 22:53:12,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 22:53:12,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 22:53:12,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 22:53:12,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 22:53:12,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 22:53:12,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 22:53:12,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 22:53:12,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 22:53:12,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 22:53:12,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 22:53:12,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 22:53:12,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 22:53:12,495 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021272054714097814); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03461508724343955, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021272054714097814); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:53:12,495 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 22:53:12,496 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014032500813793694, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021272054714097814); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03461508724343955, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021272054714097814); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 22:53:12,900 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 510 ms to try 5000 different iterations. Found a solution that moves 932 regions; Going from a computed imbalance of 0.014032500813793694 to a new imbalance of 6.390663595927033E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005645732772720048); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007813769537702511); MoveCostFunction : (multiplier=7.0, imbalance=0.10355555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005645732772720048); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:53:12,907 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv657350567:82 , srv82687239:82 , srv860786868:82 , srv871142092:82 , srv1010679784:83 , srv1778162838:83 , srv1830189018:83 , srv960677666:83 , srv1018204128:84 , srv1104560067:84 , srv1161416438:84 , srv1163317944:84 , srv1406665682:84 , srv146092503:84 , srv1522299785:84 , srv1588322877:84 , srv1653227675:84 , srv166278936:84 , srv1747142862:84 , srv2016913308:84 , srv324548761:84 , srv386397646:84 , srv598690286:84 , srv648730834:84 , srv88427127:84 , srv986395939:84 , srv1011278569:85 , srv1034434036:85 , srv1092321123:85 , srv1110969436:85 , srv1129679112:85 , srv1251876975:85 , srv1292692187:85 , srv1483045045:85 , srv1539365441:85 , srv162300567:85 , srv1675759208:85 , srv1887813980:85 , srv1915300983:85 , srv1915301776:85 , srv192478383:85 , srv1989209601:85 , srv2116673056:85 , srv358568577:85 , srv577504467:85 , srv674052822:85 , srv712731374:85 , srv717109190:85 , srv873641822:85 , srv918469565:85 , srv1059272538:86 , srv128174861:86 , srv1316272202:86 , srv1438481623:86 , srv1464790378:86 , srv1517818692:86 , srv1632175473:86 , srv1656487593:86 , srv1668213542:86 , srv1740365315:86 , srv1799238939:86 , srv1926658658:86 , srv2008655726:86 , srv401196382:86 , srv413185037:86 , srv419471270:86 , srv491810409:86 , srv491821698:86 , srv494499015:86 , srv504963305:86 , srv519989320:86 , srv607872029:86 , srv710053092:86 , srv767984058:86 , srv774708275:86 , srv931702835:86 , srv981868871:86 , srv1355890453:87 , srv138849533:87 , srv1550644895:87 , srv1633591089:87 , srv1669835438:87 , srv1759664461:87 , srv1770940651:87 , srv1794729381:87 , srv1937217598:87 , srv1948570661:87 , srv2125384353:87 , srv405698763:87 , srv523003070:87 , srv79076126:87 , srv790923500:87 , srv932504779:87 , srv1118743770:88 , srv1514039906:88 , srv210369158:88 , srv283659534:88 , srv589160946:88 , srv1323619189:89 , srv380088699:545 } 2024-02-11 22:53:12,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 22:53:12,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv405698763=66, srv1887813980=47, srv2125384353=59, srv1118743770=8, srv1915300983=48, srv2116673056=58, srv380088699=63, srv1830189018=46, srv2016913308=56, srv774708275=86, srv648730834=79, srv1483045045=23, srv607872029=78, srv146092503=21, srv88427127=93, srv657350567=80, srv1632175473=31, srv1163317944=11, srv1747142862=40, srv1059272538=4, srv1464790378=22, srv491821698=70, srv1018204128=2, srv1355890453=17, srv674052822=81, srv1104560067=6, srv1292692187=14, srv192478383=50, srv82687239=89, srv494499015=71, srv1653227675=33, srv767984058=85, srv491810409=69, srv598690286=77, srv1539365441=27, srv1011278569=1, srv1034434036=3, srv1926658658=51, srv1668213542=36, srv918469565=94, srv1550644895=28, srv138849533=18, srv2008655726=55, srv401196382=65, srv1251876975=12, srv981868871=98, srv1633591089=32, srv1669835438=37, srv717109190=84, srv413185037=67, srv871142092=91, srv1948570661=53, srv589160946=76, srv386397646=64, srv1517818692=25, srv358568577=62, srv419471270=68, srv1799238939=45, srv210369158=57, srv1406665682=19, srv1514039906=24, srv873641822=92, srv324548761=61, srv1675759208=38, srv523003070=74, srv283659534=60, srv1110969436=7, srv79076126=87, srv932504779=96, srv931702835=95, srv1656487593=34, srv1010679784=0, srv1759664461=41, srv986395939=99, srv519989320=73, srv1161416438=10, srv960677666=97, srv1092321123=5, srv1438481623=20, srv1770940651=42, srv1915301776=49, srv1129679112=9, srv166278936=35, srv710053092=82, srv712731374=83, srv1740365315=39, srv1778162838=43, srv504963305=72, srv1937217598=52, srv1522299785=26, srv1794729381=44, srv128174861=13, srv1316272202=15, srv162300567=30, srv860786868=90, srv790923500=88, srv1323619189=16, srv577504467=75, srv1588322877=29, srv1989209601=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 22:53:12,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 22:53:12,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 22:53:12,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 22:53:12,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 22:53:12,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 22:53:12,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 22:53:12,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 22:53:12,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 22:53:12,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 22:53:12,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 22:53:12,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 22:53:12,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 22:53:12,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 22:53:12,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 22:53:12,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 22:53:12,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 22:53:12,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 22:53:12,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 22:53:12,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 22:53:12,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 22:53:12,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 22:53:12,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 22:53:12,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 22:53:12,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 22:53:12,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 22:53:12,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 22:53:12,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 22:53:12,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 22:53:12,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 22:53:12,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 22:53:12,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 22:53:12,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 22:53:12,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 22:53:12,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 22:53:12,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 22:53:12,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 22:53:12,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 22:53:12,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 22:53:12,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 22:53:12,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 22:53:12,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 22:53:12,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 22:53:12,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 22:53:12,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 22:53:12,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 22:53:12,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 22:53:12,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 22:53:12,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 22:53:12,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 22:53:12,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 22:53:12,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 22:53:12,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 22:53:12,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 22:53:12,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 22:53:12,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 22:53:12,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 22:53:12,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 22:53:12,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 22:53:12,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 22:53:12,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 22:53:12,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 22:53:12,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 22:53:12,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 22:53:12,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 22:53:12,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 22:53:12,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 22:53:12,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 22:53:12,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 22:53:12,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 22:53:12,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 22:53:12,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 22:53:12,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 22:53:12,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 22:53:12,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 22:53:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 22:53:12,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 22:53:12,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 22:53:12,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 22:53:12,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 22:53:12,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 22:53:12,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 22:53:12,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 22:53:12,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 22:53:12,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 22:53:12,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 22:53:12,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 22:53:12,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 22:53:12,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 22:53:13,064 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005645732772720048); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007813769537702511); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005645732772720048); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:53:13,065 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 22:53:13,065 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.32540356109634E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005645732772720048); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007813769537702511); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005645732772720048); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 22:53:13,384 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 432 ms to try 5000 different iterations. Found a solution that moves 227 regions; Going from a computed imbalance of 6.32540356109634E-4 to a new imbalance of 2.8172100272634577E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.025222222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:53:13,385 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1522299785:83 , srv1539365441:83 , srv1747142862:83 , srv2125384353:83 , srv494499015:83 , srv790923500:83 , srv871142092:83 , srv1034434036:84 , srv1059272538:84 , srv1110969436:84 , srv1163317944:84 , srv1406665682:84 , srv1438481623:84 , srv1514039906:84 , srv1517818692:84 , srv162300567:84 , srv1633591089:84 , srv1653227675:84 , srv1675759208:84 , srv1759664461:84 , srv1794729381:84 , srv1799238939:84 , srv1915301776:84 , srv1937217598:84 , srv283659534:84 , srv386397646:84 , srv401196382:84 , srv405698763:84 , srv413185037:84 , srv419471270:84 , srv491810409:84 , srv504963305:84 , srv523003070:84 , srv577504467:84 , srv589160946:84 , srv607872029:84 , srv717109190:84 , srv767984058:84 , srv774708275:84 , srv79076126:84 , srv918469565:84 , srv931702835:84 , srv932504779:84 , srv981868871:84 , srv1010679784:85 , srv1011278569:85 , srv1018204128:85 , srv1092321123:85 , srv1104560067:85 , srv1118743770:85 , srv1129679112:85 , srv1161416438:85 , srv1251876975:85 , srv128174861:85 , srv1292692187:85 , srv1316272202:85 , srv1323619189:85 , srv1355890453:85 , srv138849533:85 , srv146092503:85 , srv1464790378:85 , srv1483045045:85 , srv1550644895:85 , srv1588322877:85 , srv1632175473:85 , srv1656487593:85 , srv166278936:85 , srv1668213542:85 , srv1669835438:85 , srv1740365315:85 , srv1770940651:85 , srv1778162838:85 , srv1830189018:85 , srv1887813980:85 , srv1915300983:85 , srv192478383:85 , srv1926658658:85 , srv1948570661:85 , srv2016913308:85 , srv210369158:85 , srv2116673056:85 , srv324548761:85 , srv358568577:85 , srv491821698:85 , srv519989320:85 , srv598690286:85 , srv648730834:85 , srv657350567:85 , srv674052822:85 , srv710053092:85 , srv712731374:85 , srv82687239:85 , srv860786868:85 , srv873641822:85 , srv88427127:85 , srv960677666:85 , srv986395939:85 , srv1989209601:86 , srv2008655726:86 , srv380088699:634 } 2024-02-11 22:53:13,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 22:53:13,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv405698763=66, srv1887813980=47, srv2125384353=59, srv1118743770=8, srv1915300983=48, srv2116673056=58, srv380088699=63, srv1830189018=46, srv2016913308=56, srv774708275=86, srv648730834=79, srv1483045045=23, srv607872029=78, srv146092503=21, srv88427127=93, srv657350567=80, srv1632175473=31, srv1163317944=11, srv1747142862=40, srv1059272538=4, srv1464790378=22, srv491821698=70, srv1018204128=2, srv1355890453=17, srv674052822=81, srv1104560067=6, srv1292692187=14, srv192478383=50, srv82687239=89, srv494499015=71, srv1653227675=33, srv767984058=85, srv491810409=69, srv598690286=77, srv1539365441=27, srv1011278569=1, srv1034434036=3, srv1926658658=51, srv1668213542=36, srv918469565=94, srv1550644895=28, srv138849533=18, srv2008655726=55, srv401196382=65, srv1251876975=12, srv981868871=98, srv1633591089=32, srv1669835438=37, srv717109190=84, srv413185037=67, srv871142092=91, srv1948570661=53, srv589160946=76, srv386397646=64, srv1517818692=25, srv358568577=62, srv419471270=68, srv1799238939=45, srv210369158=57, srv1406665682=19, srv1514039906=24, srv873641822=92, srv324548761=61, srv1675759208=38, srv523003070=74, srv283659534=60, srv1110969436=7, srv79076126=87, srv932504779=96, srv931702835=95, srv1656487593=34, srv1010679784=0, srv1759664461=41, srv986395939=99, srv519989320=73, srv1161416438=10, srv960677666=97, srv1092321123=5, srv1438481623=20, srv1770940651=42, srv1915301776=49, srv1129679112=9, srv166278936=35, srv710053092=82, srv712731374=83, srv1740365315=39, srv1778162838=43, srv504963305=72, srv1937217598=52, srv1522299785=26, srv1794729381=44, srv128174861=13, srv1316272202=15, srv162300567=30, srv860786868=90, srv790923500=88, srv1323619189=16, srv577504467=75, srv1588322877=29, srv1989209601=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 22:53:13,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 22:53:13,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 22:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 22:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 22:53:13,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 22:53:13,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 22:53:13,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 22:53:13,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 22:53:13,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 22:53:13,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 22:53:13,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 22:53:13,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 22:53:13,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 22:53:13,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 22:53:13,501 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:53:13,501 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 22:53:13,502 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6582612299826927E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 22:53:13,732 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 320 ms to try 5000 different iterations. Found a solution that moves 22 regions; Going from a computed imbalance of 2.6582612299826927E-5 to a new imbalance of 7.2272865888482504E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0024444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:53:13,732 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1104560067:83 , srv1161416438:83 , srv1323619189:83 , srv1355890453:83 , srv1915300983:83 , srv1989209601:83 , srv2008655726:83 , srv657350567:83 , srv873641822:83 , srv1010679784:84 , srv1011278569:84 , srv1018204128:84 , srv1059272538:84 , srv1092321123:84 , srv1110969436:84 , srv1118743770:84 , srv1129679112:84 , srv1163317944:84 , srv1251876975:84 , srv128174861:84 , srv1292692187:84 , srv1316272202:84 , srv138849533:84 , srv1438481623:84 , srv146092503:84 , srv1464790378:84 , srv1483045045:84 , srv1514039906:84 , srv1517818692:84 , srv1522299785:84 , srv1550644895:84 , srv1588322877:84 , srv162300567:84 , srv1632175473:84 , srv1633591089:84 , srv1653227675:84 , srv1656487593:84 , srv166278936:84 , srv1668213542:84 , srv1669835438:84 , srv1675759208:84 , srv1740365315:84 , srv1759664461:84 , srv1770940651:84 , srv1778162838:84 , srv1794729381:84 , srv1799238939:84 , srv1830189018:84 , srv1887813980:84 , srv1915301776:84 , srv192478383:84 , srv1926658658:84 , srv1937217598:84 , srv1948570661:84 , srv2016913308:84 , srv210369158:84 , srv2116673056:84 , srv2125384353:84 , srv283659534:84 , srv324548761:84 , srv358568577:84 , srv386397646:84 , srv405698763:84 , srv413185037:84 , srv419471270:84 , srv491810409:84 , srv491821698:84 , srv504963305:84 , srv519989320:84 , srv523003070:84 , srv577504467:84 , srv589160946:84 , srv598690286:84 , srv607872029:84 , srv648730834:84 , srv674052822:84 , srv710053092:84 , srv712731374:84 , srv717109190:84 , srv767984058:84 , srv774708275:84 , srv790923500:84 , srv82687239:84 , srv860786868:84 , srv88427127:84 , srv918469565:84 , srv932504779:84 , srv960677666:84 , srv981868871:84 , srv986395939:84 , srv1034434036:85 , srv1406665682:85 , srv1539365441:85 , srv1747142862:85 , srv401196382:85 , srv494499015:85 , srv79076126:85 , srv871142092:85 , srv931702835:85 , srv380088699:684 } 2024-02-11 22:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 22:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv405698763=66, srv1887813980=47, srv2125384353=59, srv1118743770=8, srv1915300983=48, srv2116673056=58, srv380088699=63, srv1830189018=46, srv2016913308=56, srv774708275=86, srv648730834=79, srv1483045045=23, srv607872029=78, srv146092503=21, srv88427127=93, srv657350567=80, srv1632175473=31, srv1163317944=11, srv1747142862=40, srv1059272538=4, srv1464790378=22, srv491821698=70, srv1018204128=2, srv1355890453=17, srv674052822=81, srv1104560067=6, srv1292692187=14, srv192478383=50, srv82687239=89, srv494499015=71, srv1653227675=33, srv767984058=85, srv491810409=69, srv598690286=77, srv1539365441=27, srv1011278569=1, srv1034434036=3, srv1926658658=51, srv1668213542=36, srv918469565=94, srv1550644895=28, srv138849533=18, srv2008655726=55, srv401196382=65, srv1251876975=12, srv981868871=98, srv1633591089=32, srv1669835438=37, srv717109190=84, srv413185037=67, srv871142092=91, srv1948570661=53, srv589160946=76, srv386397646=64, srv1517818692=25, srv358568577=62, srv419471270=68, srv1799238939=45, srv210369158=57, srv1406665682=19, srv1514039906=24, srv873641822=92, srv324548761=61, srv1675759208=38, srv523003070=74, srv283659534=60, srv1110969436=7, srv79076126=87, srv932504779=96, srv931702835=95, srv1656487593=34, srv1010679784=0, srv1759664461=41, srv986395939=99, srv519989320=73, srv1161416438=10, srv960677666=97, srv1092321123=5, srv1438481623=20, srv1770940651=42, srv1915301776=49, srv1129679112=9, srv166278936=35, srv710053092=82, srv712731374=83, srv1740365315=39, srv1778162838=43, srv504963305=72, srv1937217598=52, srv1522299785=26, srv1794729381=44, srv128174861=13, srv1316272202=15, srv162300567=30, srv860786868=90, srv790923500=88, srv1323619189=16, srv577504467=75, srv1588322877=29, srv1989209601=54} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 22:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 22:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 22:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 22:53:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 22:53:13,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 22:53:13,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 22:53:13,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 22:53:13,857 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:53:13,857 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1104560067:83 , srv1161416438:83 , srv1323619189:83 , srv1355890453:83 , srv1915300983:83 , srv1989209601:83 , srv2008655726:83 , srv657350567:83 , srv873641822:83 , srv1010679784:84 , srv1011278569:84 , srv1018204128:84 , srv1059272538:84 , srv1092321123:84 , srv1110969436:84 , srv1118743770:84 , srv1129679112:84 , srv1163317944:84 , srv1251876975:84 , srv128174861:84 , srv1292692187:84 , srv1316272202:84 , srv138849533:84 , srv1438481623:84 , srv146092503:84 , srv1464790378:84 , srv1483045045:84 , srv1514039906:84 , srv1517818692:84 , srv1522299785:84 , srv1550644895:84 , srv1588322877:84 , srv162300567:84 , srv1632175473:84 , srv1633591089:84 , srv1653227675:84 , srv1656487593:84 , srv166278936:84 , srv1668213542:84 , srv1669835438:84 , srv1675759208:84 , srv1740365315:84 , srv1759664461:84 , srv1770940651:84 , srv1778162838:84 , srv1794729381:84 , srv1799238939:84 , srv1830189018:84 , srv1887813980:84 , srv1915301776:84 , srv192478383:84 , srv1926658658:84 , srv1937217598:84 , srv1948570661:84 , srv2016913308:84 , srv210369158:84 , srv2116673056:84 , srv2125384353:84 , srv283659534:84 , srv324548761:84 , srv358568577:84 , srv386397646:84 , srv405698763:84 , srv413185037:84 , srv419471270:84 , srv491810409:84 , srv491821698:84 , srv504963305:84 , srv519989320:84 , srv523003070:84 , srv577504467:84 , srv589160946:84 , srv598690286:84 , srv607872029:84 , srv648730834:84 , srv674052822:84 , srv710053092:84 , srv712731374:84 , srv717109190:84 , srv767984058:84 , srv774708275:84 , srv790923500:84 , srv82687239:84 , srv860786868:84 , srv88427127:84 , srv918469565:84 , srv932504779:84 , srv960677666:84 , srv981868871:84 , srv986395939:84 , srv1034434036:85 , srv1406665682:85 , srv1539365441:85 , srv1747142862:85 , srv401196382:85 , srv494499015:85 , srv79076126:85 , srv871142092:85 , srv931702835:85 , srv380088699:684 } 2024-02-11 22:53:13,886 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=305 (was 271) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=2328 (was 3128) 2024-02-11 22:53:13,893 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=305, ProcessCount=170, AvailableMemoryMB=2327 2024-02-11 22:53:13,893 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-11 22:53:13,894 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-02-11 22:53:13,905 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1083271289:84 , srv1153683010:84 , srv123174950:84 , srv1237511562:84 , srv1403332904:84 , srv1416641545:84 , srv1425799063:84 , srv1434151638:84 , srv1463075012:84 , srv1575004588:84 , srv1690377349:84 , srv1747416431:84 , srv1759466010:84 , srv1837875161:84 , srv1943840934:84 , srv2035312246:84 , srv2145836806:84 , srv241084505:84 , srv265670518:84 , srv278102612:84 , srv323185566:84 , srv469907374:84 , srv49495723:84 , srv516692162:84 , srv583736479:84 , srv621588431:84 , srv676929138:84 , srv713370737:84 , srv828809226:84 , srv137194453:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-02-11 22:53:13,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 22:53:13,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv137194453=4, srv621588431=26, srv1425799063=7, srv1759466010=13, srv2145836806=17, srv123174950=2, srv265670518=19, srv1943840934=15, srv49495723=23, srv2035312246=16, srv1403332904=5, srv1747416431=12, srv583736479=25, srv469907374=22, srv713370737=28, srv323185566=21, srv828809226=29, srv1237511562=3, srv1463075012=9, srv241084505=18, srv1153683010=1, srv676929138=27, srv1083271289=0, srv1690377349=11, srv1575004588=10, srv1837875161=14, srv278102612=20, srv516692162=24, srv1416641545=6, srv1434151638=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 22:53:13,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 22:53:13,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 22:53:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 22:53:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 22:53:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 22:53:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 22:53:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 22:53:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 22:53:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 22:53:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 22:53:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 22:53:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 22:53:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 22:53:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 22:53:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 22:53:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 22:53:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 22:53:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 22:53:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 22:53:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 22:53:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 22:53:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 22:53:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 22:53:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 22:53:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 22:53:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 22:53:13,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 22:53:13,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 22:53:13,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 22:53:13,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 22:53:13,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-11 22:53:13,943 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1077529265555752, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:53:13,944 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912532569217399, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1077529265555752, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-02-11 22:55:13,931 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6409499 different iterations. Found a solution that moves 1850 regions; Going from a computed imbalance of 0.9912532569217399 to a new imbalance of 4.317992290299789E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6851851851851852, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:55:13,936 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1083271289:90 , srv1153683010:90 , srv123174950:90 , srv1237511562:90 , srv137194453:90 , srv1403332904:90 , srv1416641545:90 , srv1425799063:90 , srv1434151638:90 , srv1463075012:90 , srv1575004588:90 , srv1690377349:90 , srv1747416431:90 , srv1759466010:90 , srv1837875161:90 , srv1943840934:90 , srv2035312246:90 , srv2145836806:90 , srv241084505:90 , srv265670518:90 , srv278102612:90 , srv323185566:90 , srv469907374:90 , srv49495723:90 , srv516692162:90 , srv583736479:90 , srv621588431:90 , srv676929138:90 , srv713370737:90 , srv828809226:90 } 2024-02-11 22:55:13,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 22:55:13,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv137194453=4, srv621588431=26, srv1425799063=7, srv1759466010=13, srv2145836806=17, srv123174950=2, srv265670518=19, srv1943840934=15, srv49495723=23, srv2035312246=16, srv1403332904=5, srv1747416431=12, srv583736479=25, srv469907374=22, srv713370737=28, srv323185566=21, srv828809226=29, srv1237511562=3, srv1463075012=9, srv241084505=18, srv1153683010=1, srv676929138=27, srv1083271289=0, srv1690377349=11, srv1575004588=10, srv1837875161=14, srv278102612=20, srv516692162=24, srv1416641545=6, srv1434151638=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 22:55:13,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 22:55:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 22:55:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 22:55:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 22:55:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 22:55:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 22:55:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 22:55:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 22:55:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 22:55:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 22:55:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 22:55:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 22:55:13,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-11 22:55:13,953 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 22:55:13,953 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1083271289:90 , srv1153683010:90 , srv123174950:90 , srv1237511562:90 , srv137194453:90 , srv1403332904:90 , srv1416641545:90 , srv1425799063:90 , srv1434151638:90 , srv1463075012:90 , srv1575004588:90 , srv1690377349:90 , srv1747416431:90 , srv1759466010:90 , srv1837875161:90 , srv1943840934:90 , srv2035312246:90 , srv2145836806:90 , srv241084505:90 , srv265670518:90 , srv278102612:90 , srv323185566:90 , srv469907374:90 , srv49495723:90 , srv516692162:90 , srv583736479:90 , srv621588431:90 , srv676929138:90 , srv713370737:90 , srv828809226:90 } 2024-02-11 22:55:13,965 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=193 (was 305), ProcessCount=168 (was 170), AvailableMemoryMB=1957 (was 2327)