2024-02-07 10:53:11,500 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-02-07 10:53:11,807 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-02-07 10:53:11,918 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-02-07 10:53:11,978 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-07 10:53:11,987 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-07 10:53:12,000 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=298, ProcessCount=167, AvailableMemoryMB=3756 2024-02-07 10:53:12,002 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-07 10:53:12,002 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-07 10:53:12,134 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000450215:84 , srv1005985867:84 , srv1032726652:84 , srv106995478:84 , srv1094247521:84 , srv1143937015:84 , srv115968484:84 , srv1181243845:84 , srv1200189144:84 , srv1238777100:84 , srv1249891211:84 , srv1295053718:84 , srv1344801944:84 , srv1348220393:84 , srv1359118069:84 , srv1359447811:84 , srv1383887740:84 , srv1397921907:84 , srv1411342518:84 , srv1465620615:84 , srv148065731:84 , srv1481466765:84 , srv150339491:84 , srv1521983692:84 , srv1536057519:84 , srv1538817402:84 , srv1549932325:84 , srv15512701:84 , srv1595199264:84 , srv1598064464:84 , srv1612048972:84 , srv1617063071:84 , srv1671357468:84 , srv171972588:84 , srv1722049040:84 , srv1723461265:84 , srv1740998177:84 , srv1752143673:84 , srv178637642:84 , srv1794421183:84 , srv1801580413:84 , srv1826584828:84 , srv1839423460:84 , srv1850401510:84 , srv1857944264:84 , srv1873543030:84 , srv1898982910:84 , srv1921277258:84 , srv194101790:84 , srv196139808:84 , srv1968061715:84 , srv1990530709:84 , srv2009844819:84 , srv2062550338:84 , srv2073950289:84 , srv2074460038:84 , srv2077448534:84 , srv2081898729:84 , srv2086955673:84 , srv2090174146:84 , srv2097884971:84 , srv2099275581:84 , srv2102363626:84 , srv2109973467:84 , srv2116606657:84 , srv2118073019:84 , srv212914241:84 , srv2132480123:84 , srv2142473816:84 , srv2378154:84 , srv304346237:84 , srv321764458:84 , srv325992961:84 , srv348002488:84 , srv364519132:84 , srv397393390:84 , srv399606790:84 , srv410393862:84 , srv442164462:84 , srv47596091:84 , srv480100909:84 , srv514796621:84 , srv585228054:84 , srv600474009:84 , srv619806846:84 , srv635122149:84 , srv677086059:84 , srv68357015:84 , srv717821807:84 , srv72923343:84 , srv737837289:84 , srv746001727:84 , srv861579568:84 , srv891462974:84 , srv892971088:84 , srv919749601:84 , srv921395409:84 , srv943189993:84 , srv971733938:84 , srv1731613892:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-02-07 10:53:12,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-07 10:53:12,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2062550338=54, srv1295053718=11, srv892971088=95, srv717821807=89, srv2074460038=56, srv1249891211=10, srv2109973467=64, srv1752143673=38, srv150339491=22, srv2081898729=58, srv1740998177=37, srv1826584828=42, srv1538817402=25, srv746001727=92, srv1671357468=32, srv1348220393=13, srv1481466765=21, srv1238777100=9, srv442164462=79, srv677086059=87, srv304346237=71, srv1143937015=5, srv1465620615=19, srv1005985867=1, srv15512701=27, srv635122149=86, srv971733938=99, srv1794421183=40, srv1181243845=7, srv115968484=6, srv1857944264=45, srv106995478=3, srv921395409=97, srv1094247521=4, srv943189993=98, srv891462974=94, srv1200189144=8, srv397393390=76, srv1598064464=29, srv919749601=96, srv178637642=39, srv1595199264=28, srv1990530709=52, srv1397921907=17, srv1411342518=18, srv171972588=33, srv2118073019=66, srv72923343=90, srv1536057519=24, srv514796621=82, srv1383887740=16, srv1839423460=43, srv1722049040=34, srv321764458=72, srv1359118069=14, srv861579568=93, srv148065731=20, srv194101790=49, srv600474009=84, srv1612048972=30, srv2099275581=62, srv2132480123=68, srv1617063071=31, srv2116606657=65, srv1359447811=15, srv1000450215=0, srv619806846=85, srv2142473816=69, srv1344801944=12, srv1801580413=41, srv2090174146=60, srv1521983692=23, srv480100909=81, srv410393862=78, srv1731613892=36, srv1549932325=26, srv68357015=88, srv348002488=74, srv325992961=73, srv1723461265=35, srv1921277258=48, srv364519132=75, srv1850401510=44, srv196139808=50, srv2097884971=61, srv2102363626=63, srv1898982910=47, srv1873543030=46, srv1032726652=2, srv2073950289=55, srv399606790=77, srv2086955673=59, srv737837289=91, srv2009844819=53, srv2077448534=57, srv47596091=80, srv585228054=83, srv212914241=67, srv1968061715=51, srv2378154=70} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-07 10:53:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-07 10:53:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-07 10:53:12,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-07 10:53:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-07 10:53:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-07 10:53:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-07 10:53:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-07 10:53:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-07 10:53:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-07 10:53:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-07 10:53:12,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-07 10:53:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-07 10:53:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-07 10:53:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-07 10:53:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-07 10:53:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-07 10:53:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-07 10:53:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-07 10:53:12,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-07 10:53:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-07 10:53:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-07 10:53:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-07 10:53:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-07 10:53:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-07 10:53:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-07 10:53:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-07 10:53:12,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-07 10:53:12,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-07 10:53:12,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-07 10:53:12,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-07 10:53:12,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-07 10:53:12,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-07 10:53:12,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-07 10:53:12,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-07 10:53:12,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-07 10:53:12,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-07 10:53:12,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-07 10:53:12,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-07 10:53:12,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-07 10:53:12,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-07 10:53:12,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-07 10:53:12,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-07 10:53:12,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-07 10:53:12,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-07 10:53:12,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-07 10:53:12,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-07 10:53:12,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-07 10:53:12,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-07 10:53:12,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-07 10:53:12,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-07 10:53:12,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-07 10:53:12,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-07 10:53:12,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-07 10:53:12,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-07 10:53:12,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-07 10:53:12,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-07 10:53:12,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-07 10:53:12,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-07 10:53:12,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-07 10:53:12,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-07 10:53:12,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-07 10:53:12,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-07 10:53:12,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-07 10:53:12,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-07 10:53:12,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-07 10:53:12,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-07 10:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-07 10:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-07 10:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-07 10:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-07 10:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-07 10:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-07 10:53:12,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-07 10:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-07 10:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-07 10:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-07 10:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-07 10:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-07 10:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-07 10:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-07 10:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-07 10:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-07 10:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-07 10:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-07 10:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-07 10:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-07 10:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-07 10:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-07 10:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-07 10:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-07 10:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-07 10:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-07 10:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-07 10:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-07 10:53:12,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-07 10:53:12,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-07 10:53:12,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-07 10:53:12,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-07 10:53:12,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-07 10:53:12,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-07 10:53:12,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-07 10:53:12,508 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:53:12,508 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-07 10:53:12,509 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-07 10:53:12,879 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 586 ms to try 5000 different iterations. Found a solution that moves 4662 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18504809040639952. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04179952630176867, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05719598300038624, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.518, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04179952630176867, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18025, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2475, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:53:12,896 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv106995478:79 , srv1181243845:79 , srv410393862:79 , srv1249891211:81 , srv1723461265:81 , srv1397921907:82 , srv1598064464:82 , srv2099275581:82 , srv397393390:82 , srv746001727:82 , srv1857944264:83 , srv2009844819:83 , srv364519132:83 , srv72923343:83 , srv919749601:83 , srv1538817402:84 , srv1549932325:84 , srv171972588:84 , srv1722049040:84 , srv1752143673:84 , srv2097884971:84 , srv2102363626:84 , srv2142473816:84 , srv304346237:84 , srv325992961:84 , srv47596091:84 , srv921395409:84 , srv943189993:84 , srv1000450215:85 , srv115968484:85 , srv1238777100:85 , srv1411342518:85 , srv1465620615:85 , srv148065731:85 , srv1481466765:85 , srv1850401510:85 , srv1898982910:85 , srv2073950289:85 , srv2077448534:85 , srv2081898729:85 , srv2132480123:85 , srv619806846:85 , srv677086059:85 , srv861579568:85 , srv1005985867:86 , srv1032726652:86 , srv1348220393:86 , srv1671357468:86 , srv1990530709:86 , srv212914241:86 , srv2378154:86 , srv399606790:86 , srv68357015:86 , srv717821807:86 , srv1344801944:87 , srv1359118069:87 , srv15512701:87 , srv1595199264:87 , srv1801580413:87 , srv1826584828:87 , srv1873543030:87 , srv196139808:87 , srv1968061715:87 , srv2062550338:87 , srv2118073019:87 , srv480100909:87 , srv635122149:87 , srv737837289:87 , srv1295053718:88 , srv1359447811:88 , srv1383887740:88 , srv1536057519:88 , srv1617063071:88 , srv1740998177:88 , srv194101790:88 , srv2116606657:88 , srv321764458:88 , srv514796621:88 , srv600474009:88 , srv891462974:88 , srv150339491:89 , srv178637642:89 , srv1921277258:89 , srv2090174146:89 , srv2109973467:89 , srv585228054:89 , srv971733938:89 , srv1143937015:90 , srv1612048972:90 , srv1839423460:90 , srv442164462:90 , srv892971088:90 , srv1094247521:91 , srv1521983692:92 , srv1794421183:92 , srv2086955673:92 , srv348002488:93 , srv2074460038:95 , srv1200189144:101 , srv1731613892:461 } 2024-02-07 10:53:12,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-07 10:53:12,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2062550338=54, srv1295053718=11, srv892971088=95, srv717821807=89, srv2074460038=56, srv1249891211=10, srv2109973467=64, srv1752143673=38, srv150339491=22, srv2081898729=58, srv1740998177=37, srv1826584828=42, srv1538817402=25, srv746001727=92, srv1671357468=32, srv1348220393=13, srv1481466765=21, srv1238777100=9, srv442164462=79, srv677086059=87, srv304346237=71, srv1143937015=5, srv1465620615=19, srv1005985867=1, srv15512701=27, srv635122149=86, srv971733938=99, srv1794421183=40, srv1181243845=7, srv115968484=6, srv1857944264=45, srv106995478=3, srv921395409=97, srv1094247521=4, srv943189993=98, srv891462974=94, srv1200189144=8, srv397393390=76, srv1598064464=29, srv919749601=96, srv178637642=39, srv1595199264=28, srv1990530709=52, srv1397921907=17, srv1411342518=18, srv171972588=33, srv2118073019=66, srv72923343=90, srv1536057519=24, srv514796621=82, srv1383887740=16, srv1839423460=43, srv1722049040=34, srv321764458=72, srv1359118069=14, srv861579568=93, srv148065731=20, srv194101790=49, srv600474009=84, srv1612048972=30, srv2099275581=62, srv2132480123=68, srv1617063071=31, srv2116606657=65, srv1359447811=15, srv1000450215=0, srv619806846=85, srv2142473816=69, srv1344801944=12, srv1801580413=41, srv2090174146=60, srv1521983692=23, srv480100909=81, srv410393862=78, srv1731613892=36, srv1549932325=26, srv68357015=88, srv348002488=74, srv325992961=73, srv1723461265=35, srv1921277258=48, srv364519132=75, srv1850401510=44, srv196139808=50, srv2097884971=61, srv2102363626=63, srv1898982910=47, srv1873543030=46, srv1032726652=2, srv2073950289=55, srv399606790=77, srv2086955673=59, srv737837289=91, srv2009844819=53, srv2077448534=57, srv47596091=80, srv585228054=83, srv212914241=67, srv1968061715=51, srv2378154=70} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-07 10:53:12,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-07 10:53:12,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-07 10:53:12,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-07 10:53:12,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-07 10:53:12,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-07 10:53:12,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-07 10:53:12,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-07 10:53:12,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-07 10:53:12,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-07 10:53:12,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-07 10:53:12,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-07 10:53:12,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-07 10:53:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-07 10:53:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-07 10:53:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-07 10:53:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-07 10:53:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-07 10:53:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-07 10:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-07 10:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-07 10:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-07 10:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-07 10:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-07 10:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-07 10:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-07 10:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-07 10:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-07 10:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-07 10:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-07 10:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-07 10:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-07 10:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-07 10:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-07 10:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-07 10:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-07 10:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-07 10:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-07 10:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-07 10:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-07 10:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-07 10:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-07 10:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-07 10:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-07 10:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-07 10:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-07 10:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-07 10:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-07 10:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-07 10:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-07 10:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-07 10:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-07 10:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-07 10:53:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-07 10:53:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-07 10:53:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-07 10:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-07 10:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-07 10:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-07 10:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-07 10:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-07 10:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-07 10:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-07 10:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-07 10:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-07 10:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-07 10:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-07 10:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-07 10:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-07 10:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-07 10:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-07 10:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-07 10:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-07 10:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-07 10:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-07 10:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-07 10:53:12,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-07 10:53:12,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-07 10:53:12,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-07 10:53:12,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-07 10:53:12,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-07 10:53:12,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-07 10:53:12,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-07 10:53:12,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-07 10:53:12,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-07 10:53:12,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-07 10:53:12,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-07 10:53:12,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-07 10:53:12,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-07 10:53:12,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-07 10:53:12,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-07 10:53:12,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-07 10:53:12,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-07 10:53:12,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-07 10:53:12,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-07 10:53:12,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-07 10:53:12,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-07 10:53:12,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-07 10:53:12,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-07 10:53:12,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-07 10:53:12,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-07 10:53:12,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-07 10:53:13,115 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04179952630176867, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05719598300038624, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04179952630176867, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18025, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2475, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:53:13,115 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-07 10:53:13,115 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18501544638468487, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04179952630176867, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05719598300038624, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04179952630176867, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18025, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2475, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-07 10:53:13,373 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 374 ms to try 5000 different iterations. Found a solution that moves 2694 regions; Going from a computed imbalance of 0.18501544638468487 to a new imbalance of 0.015312692345095734. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023046443238672658); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03625882763368457, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29933333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023046443238672658); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:53:13,388 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1200189144:71 , srv2074460038:78 , srv2086955673:79 , srv1521983692:80 , srv1794421183:80 , srv1094247521:81 , srv348002488:81 , srv1143937015:82 , srv1295053718:82 , srv1536057519:82 , srv178637642:82 , srv1839423460:82 , srv1921277258:82 , srv2090174146:82 , srv514796621:82 , srv1383887740:83 , srv1595199264:83 , srv1873543030:83 , srv2062550338:83 , srv442164462:83 , srv585228054:83 , srv891462974:83 , srv971733938:83 , srv1344801944:84 , srv1359118069:84 , srv1359447811:84 , srv150339491:84 , srv1612048972:84 , srv1617063071:84 , srv1968061715:84 , srv2109973467:84 , srv2116606657:84 , srv600474009:84 , srv737837289:84 , srv892971088:84 , srv1238777100:85 , srv1348220393:85 , srv1465620615:85 , srv15512701:85 , srv1740998177:85 , srv1801580413:85 , srv1826584828:85 , srv194101790:85 , srv196139808:85 , srv1990530709:85 , srv212914241:85 , srv321764458:85 , srv635122149:85 , srv1000450215:86 , srv1005985867:86 , srv1481466765:86 , srv1671357468:86 , srv2073950289:86 , srv2118073019:86 , srv2132480123:86 , srv2378154:86 , srv325992961:86 , srv364519132:86 , srv399606790:86 , srv47596091:86 , srv619806846:86 , srv717821807:86 , srv148065731:87 , srv171972588:87 , srv1898982910:87 , srv2097884971:87 , srv2102363626:87 , srv480100909:87 , srv68357015:87 , srv72923343:87 , srv861579568:87 , srv115968484:88 , srv1411342518:88 , srv1549932325:88 , srv1722049040:88 , srv1752143673:88 , srv1850401510:88 , srv1857944264:88 , srv2081898729:88 , srv677086059:88 , srv921395409:88 , srv943189993:88 , srv1032726652:89 , srv1538817402:89 , srv2009844819:89 , srv2077448534:89 , srv304346237:89 , srv1397921907:90 , srv2142473816:90 , srv919749601:90 , srv1598064464:91 , srv1723461265:91 , srv2099275581:91 , srv746001727:91 , srv106995478:92 , srv1249891211:92 , srv397393390:92 , srv1181243845:93 , srv410393862:96 , srv1731613892:518 } 2024-02-07 10:53:13,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-07 10:53:13,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2062550338=54, srv1295053718=11, srv892971088=95, srv717821807=89, srv2074460038=56, srv1249891211=10, srv2109973467=64, srv1752143673=38, srv150339491=22, srv2081898729=58, srv1740998177=37, srv1826584828=42, srv1538817402=25, srv746001727=92, srv1671357468=32, srv1348220393=13, srv1481466765=21, srv1238777100=9, srv442164462=79, srv677086059=87, srv304346237=71, srv1143937015=5, srv1465620615=19, srv1005985867=1, srv15512701=27, srv635122149=86, srv971733938=99, srv1794421183=40, srv1181243845=7, srv115968484=6, srv1857944264=45, srv106995478=3, srv921395409=97, srv1094247521=4, srv943189993=98, srv891462974=94, srv1200189144=8, srv397393390=76, srv1598064464=29, srv919749601=96, srv178637642=39, srv1595199264=28, srv1990530709=52, srv1397921907=17, srv1411342518=18, srv171972588=33, srv2118073019=66, srv72923343=90, srv1536057519=24, srv514796621=82, srv1383887740=16, srv1839423460=43, srv1722049040=34, srv321764458=72, srv1359118069=14, srv861579568=93, srv148065731=20, srv194101790=49, srv600474009=84, srv1612048972=30, srv2099275581=62, srv2132480123=68, srv1617063071=31, srv2116606657=65, srv1359447811=15, srv1000450215=0, srv619806846=85, srv2142473816=69, srv1344801944=12, srv1801580413=41, srv2090174146=60, srv1521983692=23, srv480100909=81, srv410393862=78, srv1731613892=36, srv1549932325=26, srv68357015=88, srv348002488=74, srv325992961=73, srv1723461265=35, srv1921277258=48, srv364519132=75, srv1850401510=44, srv196139808=50, srv2097884971=61, srv2102363626=63, srv1898982910=47, srv1873543030=46, srv1032726652=2, srv2073950289=55, srv399606790=77, srv2086955673=59, srv737837289=91, srv2009844819=53, srv2077448534=57, srv47596091=80, srv585228054=83, srv212914241=67, srv1968061715=51, srv2378154=70} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-07 10:53:13,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-07 10:53:13,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-07 10:53:13,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-07 10:53:13,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-07 10:53:13,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-07 10:53:13,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-07 10:53:13,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-07 10:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-07 10:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-07 10:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-07 10:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-07 10:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-07 10:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-07 10:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-07 10:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-07 10:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-07 10:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-07 10:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-07 10:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-07 10:53:13,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-07 10:53:13,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-07 10:53:13,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-07 10:53:13,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-07 10:53:13,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-07 10:53:13,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-07 10:53:13,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-07 10:53:13,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-07 10:53:13,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-07 10:53:13,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-07 10:53:13,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-07 10:53:13,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-07 10:53:13,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-07 10:53:13,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-07 10:53:13,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-07 10:53:13,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-07 10:53:13,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-07 10:53:13,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-07 10:53:13,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-07 10:53:13,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-07 10:53:13,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-07 10:53:13,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-07 10:53:13,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-07 10:53:13,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-07 10:53:13,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-07 10:53:13,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-07 10:53:13,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-07 10:53:13,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-07 10:53:13,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-07 10:53:13,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-07 10:53:13,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-07 10:53:13,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-07 10:53:13,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-07 10:53:13,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-07 10:53:13,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-07 10:53:13,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-07 10:53:13,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-07 10:53:13,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-07 10:53:13,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-07 10:53:13,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-07 10:53:13,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-07 10:53:13,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-07 10:53:13,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-07 10:53:13,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-07 10:53:13,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-07 10:53:13,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-07 10:53:13,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-07 10:53:13,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-07 10:53:13,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-07 10:53:13,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-07 10:53:13,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-07 10:53:13,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-07 10:53:13,549 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023046443238672658); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03625882763368457, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023046443238672658); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:53:13,549 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-07 10:53:13,549 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01529382855391184, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023046443238672658); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03625882763368457, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023046443238672658); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-07 10:53:13,794 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 357 ms to try 5000 different iterations. Found a solution that moves 998 regions; Going from a computed imbalance of 0.01529382855391184 to a new imbalance of 7.527380286222328E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006894715323271239); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009960640836865495); MoveCostFunction : (multiplier=7.0, imbalance=0.11088888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006894715323271239); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:53:13,796 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1032726652:81 , srv150339491:83 , srv1612048972:83 , srv410393862:83 , srv115968484:84 , srv1249891211:84 , srv1538817402:84 , srv1598064464:84 , srv1671357468:84 , srv1826584828:84 , srv194101790:84 , srv2074460038:84 , srv2077448534:84 , srv2081898729:84 , srv2099275581:84 , srv2109973467:84 , srv2142473816:84 , srv304346237:84 , srv321764458:84 , srv348002488:84 , srv397393390:84 , srv677086059:84 , srv68357015:84 , srv746001727:84 , srv919749601:84 , srv921395409:84 , srv1005985867:85 , srv106995478:85 , srv1200189144:85 , srv1397921907:85 , srv1411342518:85 , srv148065731:85 , srv1521983692:85 , srv1549932325:85 , srv15512701:85 , srv1723461265:85 , srv1740998177:85 , srv1752143673:85 , srv1794421183:85 , srv1801580413:85 , srv1839423460:85 , srv1850401510:85 , srv1857944264:85 , srv1898982910:85 , srv2118073019:85 , srv399606790:85 , srv442164462:85 , srv480100909:85 , srv635122149:85 , srv861579568:85 , srv892971088:85 , srv943189993:85 , srv1000450215:86 , srv1094247521:86 , srv1143937015:86 , srv1238777100:86 , srv1344801944:86 , srv1359118069:86 , srv1359447811:86 , srv1383887740:86 , srv1617063071:86 , srv171972588:86 , srv1722049040:86 , srv196139808:86 , srv1968061715:86 , srv1990530709:86 , srv2062550338:86 , srv2086955673:86 , srv2090174146:86 , srv2097884971:86 , srv2102363626:86 , srv2116606657:86 , srv2378154:86 , srv585228054:86 , srv600474009:86 , srv717821807:86 , srv971733938:86 , srv1181243845:87 , srv1348220393:87 , srv1481466765:87 , srv1595199264:87 , srv178637642:87 , srv1873543030:87 , srv1921277258:87 , srv2009844819:87 , srv2073950289:87 , srv212914241:87 , srv2132480123:87 , srv514796621:87 , srv619806846:87 , srv72923343:87 , srv737837289:87 , srv891462974:87 , srv1295053718:88 , srv325992961:88 , srv364519132:88 , srv47596091:88 , srv1465620615:89 , srv1536057519:89 , srv1731613892:540 } 2024-02-07 10:53:13,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-07 10:53:13,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2062550338=54, srv1295053718=11, srv892971088=95, srv717821807=89, srv2074460038=56, srv1249891211=10, srv2109973467=64, srv1752143673=38, srv150339491=22, srv2081898729=58, srv1740998177=37, srv1826584828=42, srv1538817402=25, srv746001727=92, srv1671357468=32, srv1348220393=13, srv1481466765=21, srv1238777100=9, srv442164462=79, srv677086059=87, srv304346237=71, srv1143937015=5, srv1465620615=19, srv1005985867=1, srv15512701=27, srv635122149=86, srv971733938=99, srv1794421183=40, srv1181243845=7, srv115968484=6, srv1857944264=45, srv106995478=3, srv921395409=97, srv1094247521=4, srv943189993=98, srv891462974=94, srv1200189144=8, srv397393390=76, srv1598064464=29, srv919749601=96, srv178637642=39, srv1595199264=28, srv1990530709=52, srv1397921907=17, srv1411342518=18, srv171972588=33, srv2118073019=66, srv72923343=90, srv1536057519=24, srv514796621=82, srv1383887740=16, srv1839423460=43, srv1722049040=34, srv321764458=72, srv1359118069=14, srv861579568=93, srv148065731=20, srv194101790=49, srv600474009=84, srv1612048972=30, srv2099275581=62, srv2132480123=68, srv1617063071=31, srv2116606657=65, srv1359447811=15, srv1000450215=0, srv619806846=85, srv2142473816=69, srv1344801944=12, srv1801580413=41, srv2090174146=60, srv1521983692=23, srv480100909=81, srv410393862=78, srv1731613892=36, srv1549932325=26, srv68357015=88, srv348002488=74, srv325992961=73, srv1723461265=35, srv1921277258=48, srv364519132=75, srv1850401510=44, srv196139808=50, srv2097884971=61, srv2102363626=63, srv1898982910=47, srv1873543030=46, srv1032726652=2, srv2073950289=55, srv399606790=77, srv2086955673=59, srv737837289=91, srv2009844819=53, srv2077448534=57, srv47596091=80, srv585228054=83, srv212914241=67, srv1968061715=51, srv2378154=70} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-07 10:53:13,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-07 10:53:13,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-07 10:53:13,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-07 10:53:13,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-07 10:53:13,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-07 10:53:13,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-07 10:53:13,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-07 10:53:13,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-07 10:53:13,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-07 10:53:13,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-07 10:53:13,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-07 10:53:13,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-07 10:53:13,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-07 10:53:13,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-07 10:53:13,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-07 10:53:13,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-07 10:53:13,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-07 10:53:13,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-07 10:53:13,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-07 10:53:13,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-07 10:53:13,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-07 10:53:13,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-07 10:53:13,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-07 10:53:13,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-07 10:53:13,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-07 10:53:13,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-07 10:53:13,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-07 10:53:13,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-07 10:53:13,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-07 10:53:13,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-07 10:53:13,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-07 10:53:13,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-07 10:53:13,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-07 10:53:13,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-07 10:53:13,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-07 10:53:13,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-07 10:53:13,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-07 10:53:13,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-07 10:53:13,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-07 10:53:13,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-07 10:53:13,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-07 10:53:13,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-07 10:53:13,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-07 10:53:13,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-07 10:53:13,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-07 10:53:13,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-07 10:53:13,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-07 10:53:13,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-07 10:53:13,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-07 10:53:13,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-07 10:53:13,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-07 10:53:13,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-07 10:53:13,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-07 10:53:13,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-07 10:53:13,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-07 10:53:13,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-07 10:53:13,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-07 10:53:13,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-07 10:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-07 10:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-07 10:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-07 10:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-07 10:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-07 10:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-07 10:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-07 10:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-07 10:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-07 10:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-07 10:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-07 10:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-07 10:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-07 10:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-07 10:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-07 10:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-07 10:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-07 10:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-07 10:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-07 10:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-07 10:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-07 10:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-07 10:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-07 10:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-07 10:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-07 10:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-07 10:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-07 10:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-07 10:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-07 10:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-07 10:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-07 10:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-07 10:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-07 10:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-07 10:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-07 10:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-07 10:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-07 10:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-07 10:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-07 10:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-07 10:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-07 10:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-07 10:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-07 10:53:13,977 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006894715323271239); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009960640836865495); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006894715323271239); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:53:13,978 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-07 10:53:13,978 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.457498832616071E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006894715323271239); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009960640836865495); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006894715323271239); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-07 10:53:14,402 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 566 ms to try 5000 different iterations. Found a solution that moves 273 regions; Going from a computed imbalance of 7.457498832616071E-4 to a new imbalance of 3.581925332195173E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.030333333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:53:14,403 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1181243845:83 , srv1465620615:83 , srv1536057519:83 , srv2009844819:83 , srv2132480123:83 , srv480100909:83 , srv892971088:83 , srv1000450215:84 , srv1005985867:84 , srv1094247521:84 , srv1143937015:84 , srv1295053718:84 , srv1348220393:84 , srv1359447811:84 , srv1411342518:84 , srv1481466765:84 , srv1595199264:84 , srv1617063071:84 , srv1722049040:84 , srv1740998177:84 , srv178637642:84 , srv1850401510:84 , srv1921277258:84 , srv196139808:84 , srv2073950289:84 , srv2077448534:84 , srv2099275581:84 , srv2116606657:84 , srv2118073019:84 , srv212914241:84 , srv2142473816:84 , srv2378154:84 , srv348002488:84 , srv364519132:84 , srv397393390:84 , srv442164462:84 , srv47596091:84 , srv585228054:84 , srv600474009:84 , srv619806846:84 , srv737837289:84 , srv891462974:84 , srv971733938:84 , srv115968484:85 , srv1200189144:85 , srv1249891211:85 , srv1344801944:85 , srv1359118069:85 , srv1383887740:85 , srv1397921907:85 , srv148065731:85 , srv1521983692:85 , srv1538817402:85 , srv1549932325:85 , srv1598064464:85 , srv1612048972:85 , srv171972588:85 , srv1723461265:85 , srv1752143673:85 , srv1794421183:85 , srv1801580413:85 , srv1839423460:85 , srv1873543030:85 , srv1898982910:85 , srv194101790:85 , srv1968061715:85 , srv1990530709:85 , srv2074460038:85 , srv2081898729:85 , srv2086955673:85 , srv2090174146:85 , srv2097884971:85 , srv2102363626:85 , srv2109973467:85 , srv304346237:85 , srv321764458:85 , srv325992961:85 , srv399606790:85 , srv410393862:85 , srv514796621:85 , srv635122149:85 , srv677086059:85 , srv68357015:85 , srv717821807:85 , srv72923343:85 , srv746001727:85 , srv861579568:85 , srv919749601:85 , srv943189993:85 , srv1032726652:86 , srv106995478:86 , srv1238777100:86 , srv150339491:86 , srv15512701:86 , srv1671357468:86 , srv2062550338:86 , srv921395409:86 , srv1826584828:87 , srv1857944264:87 , srv1731613892:623 } 2024-02-07 10:53:14,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-07 10:53:14,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2062550338=54, srv1295053718=11, srv892971088=95, srv717821807=89, srv2074460038=56, srv1249891211=10, srv2109973467=64, srv1752143673=38, srv150339491=22, srv2081898729=58, srv1740998177=37, srv1826584828=42, srv1538817402=25, srv746001727=92, srv1671357468=32, srv1348220393=13, srv1481466765=21, srv1238777100=9, srv442164462=79, srv677086059=87, srv304346237=71, srv1143937015=5, srv1465620615=19, srv1005985867=1, srv15512701=27, srv635122149=86, srv971733938=99, srv1794421183=40, srv1181243845=7, srv115968484=6, srv1857944264=45, srv106995478=3, srv921395409=97, srv1094247521=4, srv943189993=98, srv891462974=94, srv1200189144=8, srv397393390=76, srv1598064464=29, srv919749601=96, srv178637642=39, srv1595199264=28, srv1990530709=52, srv1397921907=17, srv1411342518=18, srv171972588=33, srv2118073019=66, srv72923343=90, srv1536057519=24, srv514796621=82, srv1383887740=16, srv1839423460=43, srv1722049040=34, srv321764458=72, srv1359118069=14, srv861579568=93, srv148065731=20, srv194101790=49, srv600474009=84, srv1612048972=30, srv2099275581=62, srv2132480123=68, srv1617063071=31, srv2116606657=65, srv1359447811=15, srv1000450215=0, srv619806846=85, srv2142473816=69, srv1344801944=12, srv1801580413=41, srv2090174146=60, srv1521983692=23, srv480100909=81, srv410393862=78, srv1731613892=36, srv1549932325=26, srv68357015=88, srv348002488=74, srv325992961=73, srv1723461265=35, srv1921277258=48, srv364519132=75, srv1850401510=44, srv196139808=50, srv2097884971=61, srv2102363626=63, srv1898982910=47, srv1873543030=46, srv1032726652=2, srv2073950289=55, srv399606790=77, srv2086955673=59, srv737837289=91, srv2009844819=53, srv2077448534=57, srv47596091=80, srv585228054=83, srv212914241=67, srv1968061715=51, srv2378154=70} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-07 10:53:14,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-07 10:53:14,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-07 10:53:14,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-07 10:53:14,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-07 10:53:14,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-07 10:53:14,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-07 10:53:14,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-07 10:53:14,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-07 10:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-07 10:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-07 10:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-07 10:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-07 10:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-07 10:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-07 10:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-07 10:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-07 10:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-07 10:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-07 10:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-07 10:53:14,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-07 10:53:14,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-07 10:53:14,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-07 10:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-07 10:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-07 10:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-07 10:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-07 10:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-07 10:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-07 10:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-07 10:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-07 10:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-07 10:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-07 10:53:14,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-07 10:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-07 10:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-07 10:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-07 10:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-07 10:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-07 10:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-07 10:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-07 10:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-07 10:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-07 10:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-07 10:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-07 10:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-07 10:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-07 10:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-07 10:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-07 10:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-07 10:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-07 10:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-07 10:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-07 10:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-07 10:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-07 10:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-07 10:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-07 10:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-07 10:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-07 10:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-07 10:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-07 10:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-07 10:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-07 10:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-07 10:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-07 10:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-07 10:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-07 10:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-07 10:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-07 10:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-07 10:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-07 10:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-07 10:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-07 10:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-07 10:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-07 10:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-07 10:53:14,552 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:53:14,553 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-07 10:53:14,554 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.390766646478658E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-07 10:53:14,785 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 340 ms to try 5000 different iterations. Found a solution that moves 23 regions; Going from a computed imbalance of 3.390766646478658E-5 to a new imbalance of 8.420813428170152E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0025555555555555557); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:53:14,785 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv15512701:83 , srv1826584828:83 , srv1857944264:83 , srv325992961:83 , srv410393862:83 , srv717821807:83 , srv1032726652:84 , srv106995478:84 , srv1094247521:84 , srv1143937015:84 , srv115968484:84 , srv1181243845:84 , srv1200189144:84 , srv1238777100:84 , srv1249891211:84 , srv1295053718:84 , srv1344801944:84 , srv1348220393:84 , srv1359118069:84 , srv1359447811:84 , srv1383887740:84 , srv1397921907:84 , srv1411342518:84 , srv1465620615:84 , srv148065731:84 , srv1481466765:84 , srv150339491:84 , srv1521983692:84 , srv1536057519:84 , srv1538817402:84 , srv1549932325:84 , srv1598064464:84 , srv1612048972:84 , srv1617063071:84 , srv1671357468:84 , srv171972588:84 , srv1722049040:84 , srv1723461265:84 , srv1740998177:84 , srv1752143673:84 , srv178637642:84 , srv1794421183:84 , srv1801580413:84 , srv1839423460:84 , srv1850401510:84 , srv1873543030:84 , srv1898982910:84 , srv1921277258:84 , srv194101790:84 , srv196139808:84 , srv1968061715:84 , srv1990530709:84 , srv2009844819:84 , srv2062550338:84 , srv2073950289:84 , srv2074460038:84 , srv2077448534:84 , srv2081898729:84 , srv2086955673:84 , srv2090174146:84 , srv2097884971:84 , srv2102363626:84 , srv2109973467:84 , srv2116606657:84 , srv2118073019:84 , srv212914241:84 , srv2142473816:84 , srv2378154:84 , srv304346237:84 , srv321764458:84 , srv348002488:84 , srv397393390:84 , srv399606790:84 , srv442164462:84 , srv47596091:84 , srv480100909:84 , srv514796621:84 , srv585228054:84 , srv600474009:84 , srv619806846:84 , srv635122149:84 , srv677086059:84 , srv68357015:84 , srv72923343:84 , srv737837289:84 , srv746001727:84 , srv861579568:84 , srv891462974:84 , srv892971088:84 , srv919749601:84 , srv921395409:84 , srv943189993:84 , srv971733938:84 , srv1000450215:85 , srv1005985867:85 , srv1595199264:85 , srv2099275581:85 , srv2132480123:85 , srv364519132:85 , srv1731613892:684 } 2024-02-07 10:53:14,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-07 10:53:14,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2062550338=54, srv1295053718=11, srv892971088=95, srv717821807=89, srv2074460038=56, srv1249891211=10, srv2109973467=64, srv1752143673=38, srv150339491=22, srv2081898729=58, srv1740998177=37, srv1826584828=42, srv1538817402=25, srv746001727=92, srv1671357468=32, srv1348220393=13, srv1481466765=21, srv1238777100=9, srv442164462=79, srv677086059=87, srv304346237=71, srv1143937015=5, srv1465620615=19, srv1005985867=1, srv15512701=27, srv635122149=86, srv971733938=99, srv1794421183=40, srv1181243845=7, srv115968484=6, srv1857944264=45, srv106995478=3, srv921395409=97, srv1094247521=4, srv943189993=98, srv891462974=94, srv1200189144=8, srv397393390=76, srv1598064464=29, srv919749601=96, srv178637642=39, srv1595199264=28, srv1990530709=52, srv1397921907=17, srv1411342518=18, srv171972588=33, srv2118073019=66, srv72923343=90, srv1536057519=24, srv514796621=82, srv1383887740=16, srv1839423460=43, srv1722049040=34, srv321764458=72, srv1359118069=14, srv861579568=93, srv148065731=20, srv194101790=49, srv600474009=84, srv1612048972=30, srv2099275581=62, srv2132480123=68, srv1617063071=31, srv2116606657=65, srv1359447811=15, srv1000450215=0, srv619806846=85, srv2142473816=69, srv1344801944=12, srv1801580413=41, srv2090174146=60, srv1521983692=23, srv480100909=81, srv410393862=78, srv1731613892=36, srv1549932325=26, srv68357015=88, srv348002488=74, srv325992961=73, srv1723461265=35, srv1921277258=48, srv364519132=75, srv1850401510=44, srv196139808=50, srv2097884971=61, srv2102363626=63, srv1898982910=47, srv1873543030=46, srv1032726652=2, srv2073950289=55, srv399606790=77, srv2086955673=59, srv737837289=91, srv2009844819=53, srv2077448534=57, srv47596091=80, srv585228054=83, srv212914241=67, srv1968061715=51, srv2378154=70} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-07 10:53:14,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-07 10:53:14,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-07 10:53:14,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-07 10:53:14,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-07 10:53:14,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-07 10:53:14,911 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.259763985991403E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:53:14,911 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv15512701:83 , srv1826584828:83 , srv1857944264:83 , srv325992961:83 , srv410393862:83 , srv717821807:83 , srv1032726652:84 , srv106995478:84 , srv1094247521:84 , srv1143937015:84 , srv115968484:84 , srv1181243845:84 , srv1200189144:84 , srv1238777100:84 , srv1249891211:84 , srv1295053718:84 , srv1344801944:84 , srv1348220393:84 , srv1359118069:84 , srv1359447811:84 , srv1383887740:84 , srv1397921907:84 , srv1411342518:84 , srv1465620615:84 , srv148065731:84 , srv1481466765:84 , srv150339491:84 , srv1521983692:84 , srv1536057519:84 , srv1538817402:84 , srv1549932325:84 , srv1598064464:84 , srv1612048972:84 , srv1617063071:84 , srv1671357468:84 , srv171972588:84 , srv1722049040:84 , srv1723461265:84 , srv1740998177:84 , srv1752143673:84 , srv178637642:84 , srv1794421183:84 , srv1801580413:84 , srv1839423460:84 , srv1850401510:84 , srv1873543030:84 , srv1898982910:84 , srv1921277258:84 , srv194101790:84 , srv196139808:84 , srv1968061715:84 , srv1990530709:84 , srv2009844819:84 , srv2062550338:84 , srv2073950289:84 , srv2074460038:84 , srv2077448534:84 , srv2081898729:84 , srv2086955673:84 , srv2090174146:84 , srv2097884971:84 , srv2102363626:84 , srv2109973467:84 , srv2116606657:84 , srv2118073019:84 , srv212914241:84 , srv2142473816:84 , srv2378154:84 , srv304346237:84 , srv321764458:84 , srv348002488:84 , srv397393390:84 , srv399606790:84 , srv442164462:84 , srv47596091:84 , srv480100909:84 , srv514796621:84 , srv585228054:84 , srv600474009:84 , srv619806846:84 , srv635122149:84 , srv677086059:84 , srv68357015:84 , srv72923343:84 , srv737837289:84 , srv746001727:84 , srv861579568:84 , srv891462974:84 , srv892971088:84 , srv919749601:84 , srv921395409:84 , srv943189993:84 , srv971733938:84 , srv1000450215:85 , srv1005985867:85 , srv1595199264:85 , srv2099275581:85 , srv2132480123:85 , srv364519132:85 , srv1731613892:684 } 2024-02-07 10:53:14,937 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=330 (was 298) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=2856 (was 3756) 2024-02-07 10:53:14,946 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=330, ProcessCount=167, AvailableMemoryMB=2856 2024-02-07 10:53:14,947 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-07 10:53:14,947 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-02-07 10:53:14,958 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1029653303:84 , srv1113860306:84 , srv1166144237:84 , srv1177239567:84 , srv11846723:84 , srv1188821206:84 , srv1351209276:84 , srv1521334559:84 , srv1531151850:84 , srv1551790470:84 , srv159805860:84 , srv1833005583:84 , srv1914170794:84 , srv1936638415:84 , srv1967045916:84 , srv2010924192:84 , srv2133657931:84 , srv259885180:84 , srv487751215:84 , srv489984762:84 , srv508610889:84 , srv640051418:84 , srv671849637:84 , srv771380881:84 , srv7807275:84 , srv939600933:84 , srv945149300:84 , srv950936069:84 , srv962287388:84 , srv2024642816:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-02-07 10:53:14,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-07 10:53:14,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv508610889=21, srv487751215=19, srv2024642816=16, srv1177239567=3, srv1521334559=7, srv1914170794=12, srv939600933=26, srv1833005583=11, srv671849637=23, srv1029653303=0, srv159805860=10, srv7807275=25, srv945149300=27, srv1113860306=1, srv489984762=20, srv2133657931=17, srv950936069=28, srv259885180=18, srv1551790470=9, srv640051418=22, srv1967045916=14, srv1166144237=2, srv2010924192=15, srv962287388=29, srv1188821206=5, srv1531151850=8, srv1936638415=13, srv771380881=24, srv1351209276=6, srv11846723=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-07 10:53:14,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-07 10:53:14,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-07 10:53:14,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-07 10:53:14,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-07 10:53:14,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-07 10:53:14,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-07 10:53:14,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-07 10:53:14,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-07 10:53:14,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-07 10:53:14,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-07 10:53:14,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-07 10:53:14,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-07 10:53:14,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-07 10:53:14,986 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0633137416804184, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:53:14,986 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912392542796932, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0633137416804184, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-02-07 10:55:14,972 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6981140 different iterations. Found a solution that moves 1830 regions; Going from a computed imbalance of 0.9912392542796932 to a new imbalance of 4.271311292566818E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6777777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:55:14,979 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1029653303:90 , srv1113860306:90 , srv1166144237:90 , srv1177239567:90 , srv11846723:90 , srv1188821206:90 , srv1351209276:90 , srv1521334559:90 , srv1531151850:90 , srv1551790470:90 , srv159805860:90 , srv1833005583:90 , srv1914170794:90 , srv1936638415:90 , srv1967045916:90 , srv2010924192:90 , srv2024642816:90 , srv2133657931:90 , srv259885180:90 , srv487751215:90 , srv489984762:90 , srv508610889:90 , srv640051418:90 , srv671849637:90 , srv771380881:90 , srv7807275:90 , srv939600933:90 , srv945149300:90 , srv950936069:90 , srv962287388:90 } 2024-02-07 10:55:14,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-07 10:55:14,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv508610889=21, srv487751215=19, srv2024642816=16, srv1177239567=3, srv1521334559=7, srv1914170794=12, srv939600933=26, srv1833005583=11, srv671849637=23, srv1029653303=0, srv159805860=10, srv7807275=25, srv945149300=27, srv1113860306=1, srv489984762=20, srv2133657931=17, srv950936069=28, srv259885180=18, srv1551790470=9, srv640051418=22, srv1967045916=14, srv1166144237=2, srv2010924192=15, srv962287388=29, srv1188821206=5, srv1531151850=8, srv1936638415=13, srv771380881=24, srv1351209276=6, srv11846723=4} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-07 10:55:14,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-07 10:55:14,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-07 10:55:14,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-07 10:55:14,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-07 10:55:14,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-07 10:55:14,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-07 10:55:14,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-07 10:55:14,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-07 10:55:14,998 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-07 10:55:14,998 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1029653303:90 , srv1113860306:90 , srv1166144237:90 , srv1177239567:90 , srv11846723:90 , srv1188821206:90 , srv1351209276:90 , srv1521334559:90 , srv1531151850:90 , srv1551790470:90 , srv159805860:90 , srv1833005583:90 , srv1914170794:90 , srv1936638415:90 , srv1967045916:90 , srv2010924192:90 , srv2024642816:90 , srv2133657931:90 , srv259885180:90 , srv487751215:90 , srv489984762:90 , srv508610889:90 , srv640051418:90 , srv671849637:90 , srv771380881:90 , srv7807275:90 , srv939600933:90 , srv945149300:90 , srv950936069:90 , srv962287388:90 } 2024-02-07 10:55:15,019 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=177 (was 330), ProcessCount=169 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=2667 (was 2856)