2023-11-26 17:53:16,185 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-11-26 17:53:16,505 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-11-26 17:53:16,622 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-11-26 17:53:16,681 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-26 17:53:16,690 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-11-26 17:53:16,709 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=289, ProcessCount=173, AvailableMemoryMB=9728 2023-11-26 17:53:16,711 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-26 17:53:16,711 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-11-26 17:53:16,845 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1021317036:84 , srv1030880974:84 , srv1043894647:84 , srv1082696229:84 , srv1096476705:84 , srv1103120057:84 , srv1106118578:84 , srv1125876568:84 , srv1127925380:84 , srv1160922595:84 , srv1181007205:84 , srv1192877880:84 , srv1208009890:84 , srv1227303263:84 , srv1231234527:84 , srv1235674937:84 , srv1235965419:84 , srv124622740:84 , srv1288285888:84 , srv1314165534:84 , srv1341540706:84 , srv1350269561:84 , srv1387496173:84 , srv1408656358:84 , srv1411101991:84 , srv1413990611:84 , srv1418622087:84 , srv1421691530:84 , srv1452145829:84 , srv1455616901:84 , srv1469379991:84 , srv1555292406:84 , srv1579944094:84 , srv1602952820:84 , srv1609665376:84 , srv1631583356:84 , srv1643452608:84 , srv1661420090:84 , srv169092347:84 , srv1705017939:84 , srv1736419519:84 , srv1737981257:84 , srv1741603507:84 , srv1782262239:84 , srv1783097350:84 , srv1786860827:84 , srv1817577444:84 , srv1846246162:84 , srv1938713979:84 , srv197975243:84 , srv2026555183:84 , srv2050151911:84 , srv2120824036:84 , srv238838535:84 , srv280733608:84 , srv311138813:84 , srv330226050:84 , srv360717040:84 , srv368440181:84 , srv376000143:84 , srv406168381:84 , srv421385593:84 , srv422698942:84 , srv430048146:84 , srv461986770:84 , srv463715981:84 , srv478541590:84 , srv487944966:84 , srv507671892:84 , srv50909044:84 , srv539748683:84 , srv552325092:84 , srv578820503:84 , srv582401735:84 , srv586221217:84 , srv5885175:84 , srv603265224:84 , srv628033911:84 , srv64105230:84 , srv647845142:84 , srv659891702:84 , srv681239113:84 , srv686768201:84 , srv719151577:84 , srv71989305:84 , srv733177872:84 , srv743719122:84 , srv751040773:84 , srv755452978:84 , srv794384638:84 , srv79684563:84 , srv816660562:84 , srv818250671:84 , srv827037182:84 , srv843716492:84 , srv916875096:84 , srv924225021:84 , srv924926643:84 , srv951306658:84 , srv643582163:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-11-26 17:53:16,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 17:53:16,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1782262239=43, srv1737981257=41, srv50909044=69, srv794384638=90, srv1413990611=25, srv755452978=89, srv64105230=78, srv1227303263=13, srv421385593=61, srv478541590=66, srv197975243=49, srv924926643=98, srv1106118578=6, srv843716492=95, srv487944966=67, srv1938713979=48, srv681239113=82, srv1043894647=2, srv1181007205=10, srv368440181=58, srv1631583356=35, srv1387496173=22, srv360717040=57, srv1469379991=30, srv507671892=68, srv539748683=70, srv1125876568=7, srv586221217=74, srv582401735=73, srv1103120057=5, srv1817577444=46, srv124622740=17, srv376000143=59, srv311138813=55, srv1705017939=39, srv751040773=88, srv719151577=84, srv422698942=62, srv1741603507=42, srv818250671=93, srv1288285888=18, srv2120824036=52, srv71989305=85, srv1661420090=37, srv1418622087=26, srv1783097350=44, srv1235965419=16, srv659891702=81, srv628033911=77, srv1314165534=19, srv2026555183=50, srv1411101991=24, srv1452145829=28, srv1231234527=14, srv733177872=86, srv1021317036=0, srv1350269561=21, srv280733608=54, srv816660562=92, srv1127925380=8, srv1579944094=32, srv924225021=97, srv1160922595=9, srv330226050=56, srv643582163=79, srv1192877880=11, srv1030880974=1, srv406168381=60, srv1846246162=47, srv1455616901=29, srv1602952820=33, srv603265224=76, srv5885175=75, srv1786860827=45, srv430048146=63, srv1235674937=15, srv827037182=94, srv1341540706=20, srv2050151911=51, srv951306658=99, srv578820503=72, srv1643452608=36, srv169092347=38, srv1408656358=23, srv238838535=53, srv686768201=83, srv1736419519=40, srv1208009890=12, srv1421691530=27, srv647845142=80, srv743719122=87, srv1096476705=4, srv79684563=91, srv916875096=96, srv552325092=71, srv1609665376=34, srv1082696229=3, srv461986770=64, srv463715981=65, srv1555292406=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 17:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 17:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 17:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 17:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 17:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 17:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 17:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 17:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 17:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 17:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 17:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 17:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 17:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 17:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 17:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 17:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 17:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 17:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 17:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 17:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 17:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 17:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 17:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 17:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 17:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 17:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 17:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 17:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 17:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 17:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 17:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-26 17:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-26 17:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-26 17:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-26 17:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-26 17:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-26 17:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-26 17:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-26 17:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-26 17:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-26 17:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-26 17:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-26 17:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-26 17:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-26 17:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-26 17:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-26 17:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-26 17:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-26 17:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-26 17:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-26 17:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-26 17:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-26 17:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-26 17:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-26 17:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-26 17:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-26 17:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-26 17:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-26 17:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-26 17:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-26 17:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-26 17:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-26 17:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-26 17:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-26 17:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-26 17:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-26 17:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-26 17:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-26 17:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-26 17:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-26 17:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-26 17:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-26 17:53:16,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-26 17:53:16,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-26 17:53:16,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-26 17:53:16,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-26 17:53:16,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-26 17:53:16,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-26 17:53:16,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-26 17:53:16,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-26 17:53:16,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-26 17:53:16,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-26 17:53:16,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-26 17:53:16,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-26 17:53:16,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-26 17:53:16,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-26 17:53:16,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-26 17:53:16,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-26 17:53:16,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-26 17:53:16,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-26 17:53:16,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-26 17:53:16,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-26 17:53:16,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-26 17:53:16,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-26 17:53:16,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-26 17:53:16,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-26 17:53:16,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-26 17:53:16,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-26 17:53:16,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-26 17:53:16,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-26 17:53:16,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-26 17:53:17,345 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:53:17,346 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-26 17:53:17,346 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-26 17:53:17,764 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 780 ms to try 5000 different iterations. Found a solution that moves 4711 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1742979229209652. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04333318944406166, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058952700323402274, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5234444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04333318944406166, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16916666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23875, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:53:17,786 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv478541590:80 , srv1103120057:81 , srv751040773:81 , srv916875096:81 , srv280733608:82 , srv539748683:82 , srv603265224:82 , srv79684563:82 , srv827037182:82 , srv1208009890:83 , srv1341540706:83 , srv1469379991:83 , srv1737981257:83 , srv360717040:83 , srv421385593:83 , srv430048146:83 , srv5885175:83 , srv755452978:83 , srv818250671:83 , srv1021317036:84 , srv1082696229:84 , srv1127925380:84 , srv1181007205:84 , srv1455616901:84 , srv1661420090:84 , srv1782262239:84 , srv376000143:84 , srv406168381:84 , srv461986770:84 , srv586221217:84 , srv71989305:84 , srv1106118578:85 , srv1314165534:85 , srv1387496173:85 , srv1579944094:85 , srv1643452608:85 , srv1938713979:85 , srv2050151911:85 , srv330226050:85 , srv552325092:85 , srv64105230:85 , srv647845142:85 , srv733177872:85 , srv924225021:85 , srv951306658:85 , srv1096476705:86 , srv1235674937:86 , srv1288285888:86 , srv1408656358:86 , srv1418622087:86 , srv1602952820:86 , srv1741603507:86 , srv368440181:86 , srv507671892:86 , srv50909044:86 , srv659891702:86 , srv681239113:86 , srv719151577:86 , srv1231234527:87 , srv169092347:87 , srv1736419519:87 , srv1786860827:87 , srv2026555183:87 , srv2120824036:87 , srv238838535:87 , srv422698942:87 , srv578820503:87 , srv582401735:87 , srv843716492:87 , srv1043894647:88 , srv1125876568:88 , srv1192877880:88 , srv1235965419:88 , srv1350269561:88 , srv1413990611:88 , srv1609665376:88 , srv1783097350:88 , srv197975243:88 , srv311138813:88 , srv1160922595:89 , srv1452145829:89 , srv1631583356:89 , srv1817577444:89 , srv1846246162:89 , srv463715981:89 , srv1227303263:90 , srv1411101991:90 , srv1421691530:90 , srv1555292406:90 , srv487944966:90 , srv628033911:90 , srv743719122:90 , srv816660562:90 , srv1030880974:91 , srv686768201:91 , srv924926643:91 , srv794384638:92 , srv124622740:93 , srv1705017939:97 , srv643582163:475 } 2023-11-26 17:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 17:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1782262239=43, srv1737981257=41, srv50909044=69, srv794384638=90, srv1413990611=25, srv755452978=89, srv64105230=78, srv1227303263=13, srv421385593=61, srv478541590=66, srv197975243=49, srv924926643=98, srv1106118578=6, srv843716492=95, srv487944966=67, srv1938713979=48, srv681239113=82, srv1043894647=2, srv1181007205=10, srv368440181=58, srv1631583356=35, srv1387496173=22, srv360717040=57, srv1469379991=30, srv507671892=68, srv539748683=70, srv1125876568=7, srv586221217=74, srv582401735=73, srv1103120057=5, srv1817577444=46, srv124622740=17, srv376000143=59, srv311138813=55, srv1705017939=39, srv751040773=88, srv719151577=84, srv422698942=62, srv1741603507=42, srv818250671=93, srv1288285888=18, srv2120824036=52, srv71989305=85, srv1661420090=37, srv1418622087=26, srv1783097350=44, srv1235965419=16, srv659891702=81, srv628033911=77, srv1314165534=19, srv2026555183=50, srv1411101991=24, srv1452145829=28, srv1231234527=14, srv733177872=86, srv1021317036=0, srv1350269561=21, srv280733608=54, srv816660562=92, srv1127925380=8, srv1579944094=32, srv924225021=97, srv1160922595=9, srv330226050=56, srv643582163=79, srv1192877880=11, srv1030880974=1, srv406168381=60, srv1846246162=47, srv1455616901=29, srv1602952820=33, srv603265224=76, srv5885175=75, srv1786860827=45, srv430048146=63, srv1235674937=15, srv827037182=94, srv1341540706=20, srv2050151911=51, srv951306658=99, srv578820503=72, srv1643452608=36, srv169092347=38, srv1408656358=23, srv238838535=53, srv686768201=83, srv1736419519=40, srv1208009890=12, srv1421691530=27, srv647845142=80, srv743719122=87, srv1096476705=4, srv79684563=91, srv916875096=96, srv552325092=71, srv1609665376=34, srv1082696229=3, srv461986770=64, srv463715981=65, srv1555292406=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 17:53:17,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 17:53:17,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 17:53:17,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 17:53:17,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 17:53:17,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 17:53:17,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 17:53:17,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 17:53:17,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 17:53:17,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 17:53:17,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 17:53:17,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 17:53:17,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 17:53:17,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 17:53:17,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 17:53:17,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 17:53:17,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 17:53:17,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 17:53:17,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 17:53:17,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 17:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 17:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 17:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 17:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 17:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 17:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 17:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 17:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 17:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 17:53:17,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 17:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 17:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-26 17:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-26 17:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-26 17:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-26 17:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-26 17:53:17,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-26 17:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-26 17:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-26 17:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-26 17:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-26 17:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-26 17:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-26 17:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-26 17:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-26 17:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-26 17:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-26 17:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-26 17:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-26 17:53:17,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-26 17:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-26 17:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-26 17:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-26 17:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-26 17:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-26 17:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-26 17:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-26 17:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-26 17:53:17,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-26 17:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-26 17:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-26 17:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-26 17:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-26 17:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-26 17:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-26 17:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-26 17:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-26 17:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-26 17:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-26 17:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-26 17:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-26 17:53:17,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-26 17:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-26 17:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-26 17:53:17,976 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04333318944406166, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058952700323402274, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04333318944406166, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16916666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23875, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:53:17,977 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-26 17:53:17,977 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17426493579391716, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04333318944406166, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058952700323402274, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04333318944406166, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16916666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23875, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-26 17:53:18,276 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 436 ms to try 5000 different iterations. Found a solution that moves 2635 regions; Going from a computed imbalance of 0.17426493579391716 to a new imbalance of 0.012775297553060597. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023400842961379724); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035612328411725645, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2927777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023400842961379724); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042833333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:53:18,292 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1705017939:77 , srv124622740:78 , srv1030880974:79 , srv686768201:81 , srv794384638:81 , srv924926643:81 , srv1421691530:82 , srv1452145829:82 , srv1631583356:82 , srv1817577444:82 , srv1846246162:82 , srv743719122:82 , srv1043894647:83 , srv1227303263:83 , srv197975243:83 , srv487944966:83 , srv816660562:83 , srv1231234527:84 , srv1235965419:84 , srv1350269561:84 , srv1555292406:84 , srv1609665376:84 , srv1736419519:84 , srv1786860827:84 , srv2026555183:84 , srv2120824036:84 , srv238838535:84 , srv311138813:84 , srv463715981:84 , srv578820503:84 , srv582401735:84 , srv659891702:84 , srv733177872:84 , srv1096476705:85 , srv1160922595:85 , srv1235674937:85 , srv1602952820:85 , srv169092347:85 , srv2050151911:85 , srv368440181:85 , srv422698942:85 , srv507671892:85 , srv50909044:85 , srv552325092:85 , srv628033911:85 , srv1021317036:86 , srv1125876568:86 , srv1181007205:86 , srv1192877880:86 , srv1411101991:86 , srv1413990611:86 , srv1455616901:86 , srv1643452608:86 , srv1661420090:86 , srv1741603507:86 , srv1938713979:86 , srv330226050:86 , srv461986770:86 , srv681239113:86 , srv71989305:86 , srv843716492:86 , srv1127925380:87 , srv1288285888:87 , srv1314165534:87 , srv1387496173:87 , srv1469379991:87 , srv1737981257:87 , srv1782262239:87 , srv1783097350:87 , srv376000143:87 , srv5885175:87 , srv647845142:87 , srv1106118578:88 , srv1341540706:88 , srv1408656358:88 , srv1418622087:88 , srv1579944094:88 , srv406168381:88 , srv421385593:88 , srv586221217:88 , srv719151577:88 , srv755452978:88 , srv924225021:88 , srv1082696229:89 , srv1208009890:89 , srv280733608:89 , srv79684563:89 , srv818250671:89 , srv951306658:89 , srv539748683:90 , srv603265224:90 , srv916875096:90 , srv360717040:91 , srv430048146:91 , srv478541590:91 , srv64105230:91 , srv751040773:91 , srv827037182:92 , srv1103120057:93 , srv643582163:507 } 2023-11-26 17:53:18,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 17:53:18,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1782262239=43, srv1737981257=41, srv50909044=69, srv794384638=90, srv1413990611=25, srv755452978=89, srv64105230=78, srv1227303263=13, srv421385593=61, srv478541590=66, srv197975243=49, srv924926643=98, srv1106118578=6, srv843716492=95, srv487944966=67, srv1938713979=48, srv681239113=82, srv1043894647=2, srv1181007205=10, srv368440181=58, srv1631583356=35, srv1387496173=22, srv360717040=57, srv1469379991=30, srv507671892=68, srv539748683=70, srv1125876568=7, srv586221217=74, srv582401735=73, srv1103120057=5, srv1817577444=46, srv124622740=17, srv376000143=59, srv311138813=55, srv1705017939=39, srv751040773=88, srv719151577=84, srv422698942=62, srv1741603507=42, srv818250671=93, srv1288285888=18, srv2120824036=52, srv71989305=85, srv1661420090=37, srv1418622087=26, srv1783097350=44, srv1235965419=16, srv659891702=81, srv628033911=77, srv1314165534=19, srv2026555183=50, srv1411101991=24, srv1452145829=28, srv1231234527=14, srv733177872=86, srv1021317036=0, srv1350269561=21, srv280733608=54, srv816660562=92, srv1127925380=8, srv1579944094=32, srv924225021=97, srv1160922595=9, srv330226050=56, srv643582163=79, srv1192877880=11, srv1030880974=1, srv406168381=60, srv1846246162=47, srv1455616901=29, srv1602952820=33, srv603265224=76, srv5885175=75, srv1786860827=45, srv430048146=63, srv1235674937=15, srv827037182=94, srv1341540706=20, srv2050151911=51, srv951306658=99, srv578820503=72, srv1643452608=36, srv169092347=38, srv1408656358=23, srv238838535=53, srv686768201=83, srv1736419519=40, srv1208009890=12, srv1421691530=27, srv647845142=80, srv743719122=87, srv1096476705=4, srv79684563=91, srv916875096=96, srv552325092=71, srv1609665376=34, srv1082696229=3, srv461986770=64, srv463715981=65, srv1555292406=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 17:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 17:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 17:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 17:53:18,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 17:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 17:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 17:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 17:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 17:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 17:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 17:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 17:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 17:53:18,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 17:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 17:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 17:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 17:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 17:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 17:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 17:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 17:53:18,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 17:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 17:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 17:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 17:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 17:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 17:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 17:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 17:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 17:53:18,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 17:53:18,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-26 17:53:18,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-26 17:53:18,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-26 17:53:18,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-26 17:53:18,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-26 17:53:18,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-26 17:53:18,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-26 17:53:18,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-26 17:53:18,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-26 17:53:18,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-26 17:53:18,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-26 17:53:18,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-26 17:53:18,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-26 17:53:18,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-26 17:53:18,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-26 17:53:18,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-26 17:53:18,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-26 17:53:18,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-26 17:53:18,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-26 17:53:18,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-26 17:53:18,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-26 17:53:18,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-26 17:53:18,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-26 17:53:18,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-26 17:53:18,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-26 17:53:18,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-26 17:53:18,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-26 17:53:18,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-26 17:53:18,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-26 17:53:18,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-26 17:53:18,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-26 17:53:18,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-26 17:53:18,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-26 17:53:18,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-26 17:53:18,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-26 17:53:18,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-26 17:53:18,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-26 17:53:18,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-26 17:53:18,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-26 17:53:18,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-26 17:53:18,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-26 17:53:18,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-26 17:53:18,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-26 17:53:18,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-26 17:53:18,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-26 17:53:18,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-26 17:53:18,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-26 17:53:18,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-26 17:53:18,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-26 17:53:18,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-26 17:53:18,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-26 17:53:18,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-26 17:53:18,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-26 17:53:18,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-26 17:53:18,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-26 17:53:18,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-26 17:53:18,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-26 17:53:18,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-26 17:53:18,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-26 17:53:18,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-26 17:53:18,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-26 17:53:18,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-26 17:53:18,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-26 17:53:18,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-26 17:53:18,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-26 17:53:18,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-26 17:53:18,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-26 17:53:18,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-26 17:53:18,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-26 17:53:18,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-26 17:53:18,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-26 17:53:18,477 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023400842961379724); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035612328411725645, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023400842961379724); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042833333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:53:18,477 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-26 17:53:18,478 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01275684688870664, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023400842961379724); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035612328411725645, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023400842961379724); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042833333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-26 17:53:18,830 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 470 ms to try 5000 different iterations. Found a solution that moves 962 regions; Going from a computed imbalance of 0.01275684688870664 to a new imbalance of 9.751166969359404E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00800299233225351); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01156642845115122); MoveCostFunction : (multiplier=7.0, imbalance=0.1068888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00800299233225351); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:53:18,832 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1411101991:81 , srv1555292406:82 , srv64105230:82 , srv1103120057:83 , srv1125876568:83 , srv1408656358:83 , srv1418622087:83 , srv1783097350:83 , srv430048146:83 , srv628033911:83 , srv951306658:83 , srv1082696229:84 , srv1106118578:84 , srv1160922595:84 , srv1192877880:84 , srv1208009890:84 , srv1227303263:84 , srv1288285888:84 , srv1387496173:84 , srv1413990611:84 , srv1579944094:84 , srv169092347:84 , srv1705017939:84 , srv360717040:84 , srv463715981:84 , srv681239113:84 , srv719151577:84 , srv816660562:84 , srv827037182:84 , srv843716492:84 , srv1235965419:85 , srv1421691530:85 , srv1602952820:85 , srv1609665376:85 , srv1643452608:85 , srv422698942:85 , srv487944966:85 , srv586221217:85 , srv603265224:85 , srv647845142:85 , srv743719122:85 , srv794384638:85 , srv924225021:85 , srv924926643:85 , srv1021317036:86 , srv1043894647:86 , srv1096476705:86 , srv1127925380:86 , srv1231234527:86 , srv1235674937:86 , srv124622740:86 , srv1314165534:86 , srv1341540706:86 , srv1350269561:86 , srv1452145829:86 , srv1631583356:86 , srv1741603507:86 , srv1782262239:86 , srv1786860827:86 , srv1846246162:86 , srv1938713979:86 , srv197975243:86 , srv2120824036:86 , srv238838535:86 , srv280733608:86 , srv311138813:86 , srv330226050:86 , srv368440181:86 , srv376000143:86 , srv406168381:86 , srv461986770:86 , srv478541590:86 , srv507671892:86 , srv50909044:86 , srv539748683:86 , srv79684563:86 , srv818250671:86 , srv1181007205:87 , srv1469379991:87 , srv1661420090:87 , srv1736419519:87 , srv1737981257:87 , srv1817577444:87 , srv2026555183:87 , srv2050151911:87 , srv421385593:87 , srv578820503:87 , srv582401735:87 , srv5885175:87 , srv686768201:87 , srv71989305:87 , srv751040773:87 , srv916875096:87 , srv1030880974:88 , srv1455616901:88 , srv552325092:88 , srv659891702:88 , srv733177872:88 , srv755452978:88 , srv643582163:547 } 2023-11-26 17:53:18,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 17:53:18,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1782262239=43, srv1737981257=41, srv50909044=69, srv794384638=90, srv1413990611=25, srv755452978=89, srv64105230=78, srv1227303263=13, srv421385593=61, srv478541590=66, srv197975243=49, srv924926643=98, srv1106118578=6, srv843716492=95, srv487944966=67, srv1938713979=48, srv681239113=82, srv1043894647=2, srv1181007205=10, srv368440181=58, srv1631583356=35, srv1387496173=22, srv360717040=57, srv1469379991=30, srv507671892=68, srv539748683=70, srv1125876568=7, srv586221217=74, srv582401735=73, srv1103120057=5, srv1817577444=46, srv124622740=17, srv376000143=59, srv311138813=55, srv1705017939=39, srv751040773=88, srv719151577=84, srv422698942=62, srv1741603507=42, srv818250671=93, srv1288285888=18, srv2120824036=52, srv71989305=85, srv1661420090=37, srv1418622087=26, srv1783097350=44, srv1235965419=16, srv659891702=81, srv628033911=77, srv1314165534=19, srv2026555183=50, srv1411101991=24, srv1452145829=28, srv1231234527=14, srv733177872=86, srv1021317036=0, srv1350269561=21, srv280733608=54, srv816660562=92, srv1127925380=8, srv1579944094=32, srv924225021=97, srv1160922595=9, srv330226050=56, srv643582163=79, srv1192877880=11, srv1030880974=1, srv406168381=60, srv1846246162=47, srv1455616901=29, srv1602952820=33, srv603265224=76, srv5885175=75, srv1786860827=45, srv430048146=63, srv1235674937=15, srv827037182=94, srv1341540706=20, srv2050151911=51, srv951306658=99, srv578820503=72, srv1643452608=36, srv169092347=38, srv1408656358=23, srv238838535=53, srv686768201=83, srv1736419519=40, srv1208009890=12, srv1421691530=27, srv647845142=80, srv743719122=87, srv1096476705=4, srv79684563=91, srv916875096=96, srv552325092=71, srv1609665376=34, srv1082696229=3, srv461986770=64, srv463715981=65, srv1555292406=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 17:53:18,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 17:53:18,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 17:53:18,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 17:53:18,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 17:53:18,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 17:53:18,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 17:53:18,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 17:53:18,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 17:53:18,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 17:53:18,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 17:53:18,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 17:53:18,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 17:53:18,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 17:53:18,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 17:53:18,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 17:53:18,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 17:53:18,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 17:53:18,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 17:53:18,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 17:53:18,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 17:53:18,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 17:53:18,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 17:53:18,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 17:53:18,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 17:53:18,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 17:53:18,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 17:53:18,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 17:53:18,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 17:53:18,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 17:53:18,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 17:53:18,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-26 17:53:18,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-26 17:53:18,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-26 17:53:18,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-26 17:53:18,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-26 17:53:18,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-26 17:53:18,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-26 17:53:18,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-26 17:53:18,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-26 17:53:18,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-26 17:53:18,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-26 17:53:18,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-26 17:53:18,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-26 17:53:18,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-26 17:53:18,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-26 17:53:18,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-26 17:53:18,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-26 17:53:18,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-26 17:53:18,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-26 17:53:18,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-26 17:53:18,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-26 17:53:18,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-26 17:53:18,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-26 17:53:18,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-26 17:53:18,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-26 17:53:18,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-26 17:53:18,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-26 17:53:18,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-26 17:53:18,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-26 17:53:18,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-26 17:53:18,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-26 17:53:18,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-26 17:53:18,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-26 17:53:18,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-26 17:53:18,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-26 17:53:18,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-26 17:53:18,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-26 17:53:18,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-26 17:53:18,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-26 17:53:18,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-26 17:53:18,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-26 17:53:18,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-26 17:53:18,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-26 17:53:18,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-26 17:53:18,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-26 17:53:18,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-26 17:53:18,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-26 17:53:18,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-26 17:53:18,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-26 17:53:18,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-26 17:53:18,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-26 17:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-26 17:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-26 17:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-26 17:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-26 17:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-26 17:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-26 17:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-26 17:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-26 17:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-26 17:53:18,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-26 17:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-26 17:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-26 17:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-26 17:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-26 17:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-26 17:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-26 17:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-26 17:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-26 17:53:18,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-26 17:53:18,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-26 17:53:19,024 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00800299233225351); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01156642845115122); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00800299233225351); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:53:19,025 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-26 17:53:19,027 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.683806289630728E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00800299233225351); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01156642845115122); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00800299233225351); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-26 17:53:19,317 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 432 ms to try 5000 different iterations. Found a solution that moves 270 regions; Going from a computed imbalance of 9.683806289630728E-4 to a new imbalance of 3.559704955620138E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.03, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:53:19,318 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv686768201:82 , srv1314165534:83 , srv582401735:83 , srv751040773:83 , srv755452978:83 , srv827037182:83 , srv1030880974:84 , srv1082696229:84 , srv1160922595:84 , srv1192877880:84 , srv1350269561:84 , srv1408656358:84 , srv1413990611:84 , srv1455616901:84 , srv1705017939:84 , srv1736419519:84 , srv1741603507:84 , srv1783097350:84 , srv1817577444:84 , srv2026555183:84 , srv311138813:84 , srv360717040:84 , srv406168381:84 , srv421385593:84 , srv422698942:84 , srv487944966:84 , srv539748683:84 , srv552325092:84 , srv628033911:84 , srv64105230:84 , srv659891702:84 , srv719151577:84 , srv794384638:84 , srv818250671:84 , srv916875096:84 , srv924225021:84 , srv1043894647:85 , srv1096476705:85 , srv1103120057:85 , srv1106118578:85 , srv1125876568:85 , srv1127925380:85 , srv1181007205:85 , srv1227303263:85 , srv1235674937:85 , srv124622740:85 , srv1288285888:85 , srv1341540706:85 , srv1411101991:85 , srv1421691530:85 , srv1452145829:85 , srv1469379991:85 , srv1555292406:85 , srv1579944094:85 , srv1609665376:85 , srv1631583356:85 , srv1661420090:85 , srv1737981257:85 , srv1782262239:85 , srv1786860827:85 , srv1846246162:85 , srv1938713979:85 , srv197975243:85 , srv2050151911:85 , srv238838535:85 , srv280733608:85 , srv330226050:85 , srv368440181:85 , srv376000143:85 , srv430048146:85 , srv463715981:85 , srv478541590:85 , srv50909044:85 , srv578820503:85 , srv586221217:85 , srv5885175:85 , srv603265224:85 , srv647845142:85 , srv71989305:85 , srv733177872:85 , srv743719122:85 , srv79684563:85 , srv816660562:85 , srv843716492:85 , srv924926643:85 , srv951306658:85 , srv1021317036:86 , srv1231234527:86 , srv1235965419:86 , srv1387496173:86 , srv1418622087:86 , srv1602952820:86 , srv1643452608:86 , srv169092347:86 , srv2120824036:86 , srv461986770:86 , srv507671892:86 , srv681239113:86 , srv1208009890:87 , srv643582163:614 } 2023-11-26 17:53:19,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 17:53:19,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1782262239=43, srv1737981257=41, srv50909044=69, srv794384638=90, srv1413990611=25, srv755452978=89, srv64105230=78, srv1227303263=13, srv421385593=61, srv478541590=66, srv197975243=49, srv924926643=98, srv1106118578=6, srv843716492=95, srv487944966=67, srv1938713979=48, srv681239113=82, srv1043894647=2, srv1181007205=10, srv368440181=58, srv1631583356=35, srv1387496173=22, srv360717040=57, srv1469379991=30, srv507671892=68, srv539748683=70, srv1125876568=7, srv586221217=74, srv582401735=73, srv1103120057=5, srv1817577444=46, srv124622740=17, srv376000143=59, srv311138813=55, srv1705017939=39, srv751040773=88, srv719151577=84, srv422698942=62, srv1741603507=42, srv818250671=93, srv1288285888=18, srv2120824036=52, srv71989305=85, srv1661420090=37, srv1418622087=26, srv1783097350=44, srv1235965419=16, srv659891702=81, srv628033911=77, srv1314165534=19, srv2026555183=50, srv1411101991=24, srv1452145829=28, srv1231234527=14, srv733177872=86, srv1021317036=0, srv1350269561=21, srv280733608=54, srv816660562=92, srv1127925380=8, srv1579944094=32, srv924225021=97, srv1160922595=9, srv330226050=56, srv643582163=79, srv1192877880=11, srv1030880974=1, srv406168381=60, srv1846246162=47, srv1455616901=29, srv1602952820=33, srv603265224=76, srv5885175=75, srv1786860827=45, srv430048146=63, srv1235674937=15, srv827037182=94, srv1341540706=20, srv2050151911=51, srv951306658=99, srv578820503=72, srv1643452608=36, srv169092347=38, srv1408656358=23, srv238838535=53, srv686768201=83, srv1736419519=40, srv1208009890=12, srv1421691530=27, srv647845142=80, srv743719122=87, srv1096476705=4, srv79684563=91, srv916875096=96, srv552325092=71, srv1609665376=34, srv1082696229=3, srv461986770=64, srv463715981=65, srv1555292406=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 17:53:19,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 17:53:19,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 17:53:19,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 17:53:19,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 17:53:19,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-26 17:53:19,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-26 17:53:19,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-26 17:53:19,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-26 17:53:19,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-26 17:53:19,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-26 17:53:19,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-26 17:53:19,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-26 17:53:19,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-26 17:53:19,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-26 17:53:19,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-26 17:53:19,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-26 17:53:19,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-26 17:53:19,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-26 17:53:19,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-26 17:53:19,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-26 17:53:19,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-26 17:53:19,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-26 17:53:19,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-26 17:53:19,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-26 17:53:19,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-26 17:53:19,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-26 17:53:19,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-26 17:53:19,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-26 17:53:19,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-26 17:53:19,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-26 17:53:19,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-26 17:53:19,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-26 17:53:19,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-26 17:53:19,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-26 17:53:19,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-26 17:53:19,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-26 17:53:19,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-26 17:53:19,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-26 17:53:19,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-26 17:53:19,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-26 17:53:19,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-26 17:53:19,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-26 17:53:19,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-26 17:53:19,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-26 17:53:19,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-26 17:53:19,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-26 17:53:19,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-26 17:53:19,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-26 17:53:19,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-26 17:53:19,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-26 17:53:19,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-26 17:53:19,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-26 17:53:19,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-26 17:53:19,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-26 17:53:19,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-26 17:53:19,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-26 17:53:19,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-26 17:53:19,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-26 17:53:19,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-26 17:53:19,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-26 17:53:19,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-26 17:53:19,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-26 17:53:19,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-26 17:53:19,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-26 17:53:19,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-26 17:53:19,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-26 17:53:19,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-26 17:53:19,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-26 17:53:19,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-26 17:53:19,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-26 17:53:19,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-26 17:53:19,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-26 17:53:19,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-26 17:53:19,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-26 17:53:19,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-26 17:53:19,483 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:53:19,484 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-26 17:53:19,484 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.370646914801607E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-26 17:53:19,780 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 417 ms to try 5000 different iterations. Found a solution that moves 30 regions; Going from a computed imbalance of 3.370646914801607E-5 to a new imbalance of 6.9541335436303165E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0033333333333333335); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:53:19,780 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1208009890:83 , srv1231234527:83 , srv1235965419:83 , srv1418622087:83 , srv2120824036:83 , srv507671892:83 , srv578820503:83 , srv1021317036:84 , srv1030880974:84 , srv1043894647:84 , srv1096476705:84 , srv1103120057:84 , srv1106118578:84 , srv1125876568:84 , srv1127925380:84 , srv1160922595:84 , srv1181007205:84 , srv1192877880:84 , srv1227303263:84 , srv1235674937:84 , srv124622740:84 , srv1288285888:84 , srv1341540706:84 , srv1350269561:84 , srv1387496173:84 , srv1411101991:84 , srv1413990611:84 , srv1421691530:84 , srv1452145829:84 , srv1455616901:84 , srv1469379991:84 , srv1579944094:84 , srv1602952820:84 , srv1609665376:84 , srv1631583356:84 , srv1643452608:84 , srv1661420090:84 , srv169092347:84 , srv1705017939:84 , srv1736419519:84 , srv1737981257:84 , srv1741603507:84 , srv1782262239:84 , srv1783097350:84 , srv1786860827:84 , srv1817577444:84 , srv1846246162:84 , srv1938713979:84 , srv197975243:84 , srv2026555183:84 , srv2050151911:84 , srv238838535:84 , srv280733608:84 , srv311138813:84 , srv330226050:84 , srv360717040:84 , srv368440181:84 , srv376000143:84 , srv406168381:84 , srv421385593:84 , srv430048146:84 , srv461986770:84 , srv463715981:84 , srv478541590:84 , srv487944966:84 , srv50909044:84 , srv539748683:84 , srv552325092:84 , srv586221217:84 , srv5885175:84 , srv603265224:84 , srv628033911:84 , srv64105230:84 , srv647845142:84 , srv659891702:84 , srv681239113:84 , srv719151577:84 , srv71989305:84 , srv733177872:84 , srv743719122:84 , srv751040773:84 , srv755452978:84 , srv794384638:84 , srv79684563:84 , srv816660562:84 , srv818250671:84 , srv843716492:84 , srv916875096:84 , srv924225021:84 , srv924926643:84 , srv951306658:84 , srv1082696229:85 , srv1314165534:85 , srv1408656358:85 , srv1555292406:85 , srv422698942:85 , srv582401735:85 , srv686768201:85 , srv827037182:85 , srv643582163:683 } 2023-11-26 17:53:19,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 17:53:19,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1782262239=43, srv1737981257=41, srv50909044=69, srv794384638=90, srv1413990611=25, srv755452978=89, srv64105230=78, srv1227303263=13, srv421385593=61, srv478541590=66, srv197975243=49, srv924926643=98, srv1106118578=6, srv843716492=95, srv487944966=67, srv1938713979=48, srv681239113=82, srv1043894647=2, srv1181007205=10, srv368440181=58, srv1631583356=35, srv1387496173=22, srv360717040=57, srv1469379991=30, srv507671892=68, srv539748683=70, srv1125876568=7, srv586221217=74, srv582401735=73, srv1103120057=5, srv1817577444=46, srv124622740=17, srv376000143=59, srv311138813=55, srv1705017939=39, srv751040773=88, srv719151577=84, srv422698942=62, srv1741603507=42, srv818250671=93, srv1288285888=18, srv2120824036=52, srv71989305=85, srv1661420090=37, srv1418622087=26, srv1783097350=44, srv1235965419=16, srv659891702=81, srv628033911=77, srv1314165534=19, srv2026555183=50, srv1411101991=24, srv1452145829=28, srv1231234527=14, srv733177872=86, srv1021317036=0, srv1350269561=21, srv280733608=54, srv816660562=92, srv1127925380=8, srv1579944094=32, srv924225021=97, srv1160922595=9, srv330226050=56, srv643582163=79, srv1192877880=11, srv1030880974=1, srv406168381=60, srv1846246162=47, srv1455616901=29, srv1602952820=33, srv603265224=76, srv5885175=75, srv1786860827=45, srv430048146=63, srv1235674937=15, srv827037182=94, srv1341540706=20, srv2050151911=51, srv951306658=99, srv578820503=72, srv1643452608=36, srv169092347=38, srv1408656358=23, srv238838535=53, srv686768201=83, srv1736419519=40, srv1208009890=12, srv1421691530=27, srv647845142=80, srv743719122=87, srv1096476705=4, srv79684563=91, srv916875096=96, srv552325092=71, srv1609665376=34, srv1082696229=3, srv461986770=64, srv463715981=65, srv1555292406=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 17:53:19,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 17:53:19,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 17:53:19,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 17:53:19,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 17:53:19,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 17:53:19,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 17:53:19,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 17:53:19,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 17:53:19,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 17:53:19,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 17:53:19,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 17:53:19,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 17:53:19,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 17:53:19,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 17:53:19,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 17:53:19,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 17:53:19,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 17:53:19,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-26 17:53:19,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-26 17:53:19,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-26 17:53:19,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-26 17:53:19,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-26 17:53:19,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-26 17:53:19,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-26 17:53:19,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-26 17:53:19,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-26 17:53:19,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-26 17:53:19,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-26 17:53:19,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-26 17:53:19,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-26 17:53:19,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-26 17:53:19,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-26 17:53:19,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-26 17:53:19,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-26 17:53:19,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-26 17:53:19,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-26 17:53:19,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-26 17:53:19,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-26 17:53:19,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-26 17:53:19,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-26 17:53:19,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-26 17:53:19,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-26 17:53:19,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-26 17:53:19,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-26 17:53:19,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-26 17:53:19,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-26 17:53:19,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-26 17:53:19,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-26 17:53:19,922 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:53:19,922 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1208009890:83 , srv1231234527:83 , srv1235965419:83 , srv1418622087:83 , srv2120824036:83 , srv507671892:83 , srv578820503:83 , srv1021317036:84 , srv1030880974:84 , srv1043894647:84 , srv1096476705:84 , srv1103120057:84 , srv1106118578:84 , srv1125876568:84 , srv1127925380:84 , srv1160922595:84 , srv1181007205:84 , srv1192877880:84 , srv1227303263:84 , srv1235674937:84 , srv124622740:84 , srv1288285888:84 , srv1341540706:84 , srv1350269561:84 , srv1387496173:84 , srv1411101991:84 , srv1413990611:84 , srv1421691530:84 , srv1452145829:84 , srv1455616901:84 , srv1469379991:84 , srv1579944094:84 , srv1602952820:84 , srv1609665376:84 , srv1631583356:84 , srv1643452608:84 , srv1661420090:84 , srv169092347:84 , srv1705017939:84 , srv1736419519:84 , srv1737981257:84 , srv1741603507:84 , srv1782262239:84 , srv1783097350:84 , srv1786860827:84 , srv1817577444:84 , srv1846246162:84 , srv1938713979:84 , srv197975243:84 , srv2026555183:84 , srv2050151911:84 , srv238838535:84 , srv280733608:84 , srv311138813:84 , srv330226050:84 , srv360717040:84 , srv368440181:84 , srv376000143:84 , srv406168381:84 , srv421385593:84 , srv430048146:84 , srv461986770:84 , srv463715981:84 , srv478541590:84 , srv487944966:84 , srv50909044:84 , srv539748683:84 , srv552325092:84 , srv586221217:84 , srv5885175:84 , srv603265224:84 , srv628033911:84 , srv64105230:84 , srv647845142:84 , srv659891702:84 , srv681239113:84 , srv719151577:84 , srv71989305:84 , srv733177872:84 , srv743719122:84 , srv751040773:84 , srv755452978:84 , srv794384638:84 , srv79684563:84 , srv816660562:84 , srv818250671:84 , srv843716492:84 , srv916875096:84 , srv924225021:84 , srv924926643:84 , srv951306658:84 , srv1082696229:85 , srv1314165534:85 , srv1408656358:85 , srv1555292406:85 , srv422698942:85 , srv582401735:85 , srv686768201:85 , srv827037182:85 , srv643582163:683 } 2023-11-26 17:53:19,959 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=289 (was 289), ProcessCount=173 (was 173), AvailableMemoryMB=8798 (was 9728) 2023-11-26 17:53:19,967 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=289, ProcessCount=173, AvailableMemoryMB=8797 2023-11-26 17:53:19,967 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-26 17:53:19,968 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-11-26 17:53:19,984 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1027059164:84 , srv1108549784:84 , srv1134420725:84 , srv124270455:84 , srv1348904971:84 , srv1393480139:84 , srv1447089204:84 , srv1491444265:84 , srv1497097461:84 , srv1530210887:84 , srv1626024722:84 , srv1642087451:84 , srv1748257639:84 , srv1874061529:84 , srv1906652146:84 , srv2002207385:84 , srv2062410649:84 , srv2105025590:84 , srv2113222281:84 , srv2124570165:84 , srv329217431:84 , srv346859310:84 , srv435881341:84 , srv465972062:84 , srv468437353:84 , srv58128040:84 , srv611412675:84 , srv79079950:84 , srv8103298:84 , srv1331659281:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-11-26 17:53:19,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 17:53:19,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1530210887=10, srv58128040=26, srv2002207385=16, srv611412675=27, srv124270455=3, srv465972062=24, srv8103298=29, srv1626024722=11, srv2105025590=18, srv1497097461=9, srv1027059164=0, srv1491444265=8, srv346859310=22, srv1108549784=1, srv329217431=21, srv1447089204=7, srv1331659281=4, srv2113222281=19, srv1393480139=6, srv1874061529=14, srv79079950=28, srv2062410649=17, srv1642087451=12, srv435881341=23, srv2124570165=20, srv1906652146=15, srv1748257639=13, srv1134420725=2, srv1348904971=5, srv468437353=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 17:53:20,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 17:53:20,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 17:53:20,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 17:53:20,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 17:53:20,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 17:53:20,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 17:53:20,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 17:53:20,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 17:53:20,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 17:53:20,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 17:53:20,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 17:53:20,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 17:53:20,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 17:53:20,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 17:53:20,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 17:53:20,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 17:53:20,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 17:53:20,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 17:53:20,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 17:53:20,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 17:53:20,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 17:53:20,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 17:53:20,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 17:53:20,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 17:53:20,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 17:53:20,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 17:53:20,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 17:53:20,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 17:53:20,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 17:53:20,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 17:53:20,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-11-26 17:53:20,033 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0237958388979305, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:53:20,033 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912268023175643, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0237958388979305, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-11-26 17:55:20,008 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6186898 different iterations. Found a solution that moves 1847 regions; Going from a computed imbalance of 0.9912268023175643 to a new imbalance of 4.3109901406398434E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6840740740740741, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:55:20,013 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1027059164:90 , srv1108549784:90 , srv1134420725:90 , srv124270455:90 , srv1331659281:90 , srv1348904971:90 , srv1393480139:90 , srv1447089204:90 , srv1491444265:90 , srv1497097461:90 , srv1530210887:90 , srv1626024722:90 , srv1642087451:90 , srv1748257639:90 , srv1874061529:90 , srv1906652146:90 , srv2002207385:90 , srv2062410649:90 , srv2105025590:90 , srv2113222281:90 , srv2124570165:90 , srv329217431:90 , srv346859310:90 , srv435881341:90 , srv465972062:90 , srv468437353:90 , srv58128040:90 , srv611412675:90 , srv79079950:90 , srv8103298:90 } 2023-11-26 17:55:20,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 17:55:20,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1530210887=10, srv58128040=26, srv2002207385=16, srv611412675=27, srv124270455=3, srv465972062=24, srv8103298=29, srv1626024722=11, srv2105025590=18, srv1497097461=9, srv1027059164=0, srv1491444265=8, srv346859310=22, srv1108549784=1, srv329217431=21, srv1447089204=7, srv1331659281=4, srv2113222281=19, srv1393480139=6, srv1874061529=14, srv79079950=28, srv2062410649=17, srv1642087451=12, srv435881341=23, srv2124570165=20, srv1906652146=15, srv1748257639=13, srv1134420725=2, srv1348904971=5, srv468437353=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 17:55:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 17:55:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 17:55:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 17:55:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 17:55:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 17:55:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 17:55:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 17:55:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 17:55:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 17:55:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 17:55:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 17:55:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 17:55:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-11-26 17:55:20,031 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 17:55:20,031 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1027059164:90 , srv1108549784:90 , srv1134420725:90 , srv124270455:90 , srv1331659281:90 , srv1348904971:90 , srv1393480139:90 , srv1447089204:90 , srv1491444265:90 , srv1497097461:90 , srv1530210887:90 , srv1626024722:90 , srv1642087451:90 , srv1748257639:90 , srv1874061529:90 , srv1906652146:90 , srv2002207385:90 , srv2062410649:90 , srv2105025590:90 , srv2113222281:90 , srv2124570165:90 , srv329217431:90 , srv346859310:90 , srv435881341:90 , srv465972062:90 , srv468437353:90 , srv58128040:90 , srv611412675:90 , srv79079950:90 , srv8103298:90 } 2023-11-26 17:55:20,045 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=162 (was 289), ProcessCount=171 (was 173), AvailableMemoryMB=8314 (was 8797)