2023-11-26 15:53:14,676 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-11-26 15:53:14,992 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-11-26 15:53:15,112 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-11-26 15:53:15,179 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-26 15:53:15,187 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-11-26 15:53:15,200 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=401, ProcessCount=172, AvailableMemoryMB=9748 2023-11-26 15:53:15,202 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-26 15:53:15,203 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-11-26 15:53:15,328 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1014040807:84 , srv1054912049:84 , srv1058948247:84 , srv1059234387:84 , srv1083132618:84 , srv1088908861:84 , srv1092094448:84 , srv1097281304:84 , srv1105549297:84 , srv1124706291:84 , srv1143979237:84 , srv1172173830:84 , srv1181748615:84 , srv1194428226:84 , srv1205105261:84 , srv1231033875:84 , srv1243918645:84 , srv1253896178:84 , srv1272836954:84 , srv1291767190:84 , srv1298541050:84 , srv1354676898:84 , srv1378381759:84 , srv1399165214:84 , srv1431258769:84 , srv144270541:84 , srv1465220767:84 , srv1503581201:84 , srv1506312447:84 , srv1536019568:84 , srv153870199:84 , srv1566945933:84 , srv1567763052:84 , srv1575467824:84 , srv1598949683:84 , srv1622400003:84 , srv1670624263:84 , srv1688931384:84 , srv1710272573:84 , srv1723594481:84 , srv1737156403:84 , srv1782923150:84 , srv178788037:84 , srv1790325855:84 , srv1801315842:84 , srv181809038:84 , srv1879375683:84 , srv1882056138:84 , srv1895419594:84 , srv1918221241:84 , srv1923081264:84 , srv1927038036:84 , srv1943697046:84 , srv1956235166:84 , srv1970814084:84 , srv2017248143:84 , srv2019898724:84 , srv2026219013:84 , srv2057735261:84 , srv2058504092:84 , srv2061044220:84 , srv2095005539:84 , srv2113716144:84 , srv2117303428:84 , srv219533460:84 , srv283639093:84 , srv291204262:84 , srv324600116:84 , srv341062472:84 , srv375457675:84 , srv37972116:84 , srv399340776:84 , srv409739958:84 , srv419230548:84 , srv460165825:84 , srv477170496:84 , srv483834427:84 , srv54728020:84 , srv556105946:84 , srv578592149:84 , srv590065965:84 , srv646016540:84 , srv654589811:84 , srv656617629:84 , srv682747403:84 , srv692818989:84 , srv699527273:84 , srv722885846:84 , srv790062514:84 , srv829234529:84 , srv85235469:84 , srv860683481:84 , srv869475352:84 , srv874021190:84 , srv880748095:84 , srv918184816:84 , srv951791160:84 , srv972460637:84 , srv997613201:84 , srv859917738:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-11-26 15:53:15,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 15:53:15,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1172173830=11, srv1567763052=32, srv291204262=66, srv1895419594=48, srv1059234387=3, srv2113716144=62, srv1923081264=50, srv1181748615=12, srv2019898724=56, srv722885846=87, srv144270541=25, srv1058948247=2, srv874021190=94, srv219533460=64, srv1882056138=47, srv1927038036=51, srv1231033875=15, srv1737156403=40, srv181809038=45, srv2058504092=59, srv880748095=95, srv860683481=92, srv2117303428=63, srv1956235166=53, srv859917738=91, srv1088908861=5, srv1622400003=35, srv1723594481=39, srv1943697046=52, srv153870199=30, srv1205105261=14, srv1566945933=31, srv1536019568=29, srv1105549297=8, srv1431258769=24, srv1194428226=13, srv1575467824=33, srv1503581201=27, srv682747403=84, srv1083132618=4, srv869475352=93, srv1143979237=10, srv1670624263=36, srv1465220767=26, srv1272836954=18, srv283639093=65, srv1782923150=41, srv341062472=68, srv375457675=69, srv2095005539=61, srv460165825=74, srv829234529=89, srv409739958=72, srv918184816=96, srv1054912049=1, srv1124706291=9, srv178788037=42, srv1918221241=49, srv1378381759=22, srv37972116=70, srv590065965=80, srv656617629=83, srv1291767190=19, srv1506312447=28, srv1014040807=0, srv1879375683=46, srv1399165214=23, srv419230548=73, srv85235469=90, srv1092094448=6, srv1710272573=38, srv1688931384=37, srv477170496=75, srv1097281304=7, srv324600116=67, srv399340776=71, srv1253896178=17, srv1790325855=43, srv2026219013=57, srv699527273=86, srv2061044220=60, srv654589811=82, srv1243918645=16, srv692818989=85, srv790062514=88, srv1801315842=44, srv997613201=99, srv1354676898=21, srv1298541050=20, srv483834427=76, srv646016540=81, srv2057735261=58, srv54728020=77, srv578592149=79, srv972460637=98, srv556105946=78, srv1598949683=34, srv1970814084=54, srv951791160=97, srv2017248143=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 15:53:15,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 15:53:15,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 15:53:15,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 15:53:15,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 15:53:15,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 15:53:15,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 15:53:15,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 15:53:15,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 15:53:15,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 15:53:15,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 15:53:15,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 15:53:15,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 15:53:15,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 15:53:15,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 15:53:15,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 15:53:15,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 15:53:15,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 15:53:15,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 15:53:15,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 15:53:15,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 15:53:15,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 15:53:15,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 15:53:15,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 15:53:15,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 15:53:15,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 15:53:15,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 15:53:15,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 15:53:15,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 15:53:15,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 15:53:15,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 15:53:15,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-26 15:53:15,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-26 15:53:15,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-26 15:53:15,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-26 15:53:15,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-26 15:53:15,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-26 15:53:15,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-26 15:53:15,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-26 15:53:15,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-26 15:53:15,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-26 15:53:15,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-26 15:53:15,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-26 15:53:15,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-26 15:53:15,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-26 15:53:15,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-26 15:53:15,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-26 15:53:15,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-26 15:53:15,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-26 15:53:15,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-26 15:53:15,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-26 15:53:15,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-26 15:53:15,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-26 15:53:15,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-26 15:53:15,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-26 15:53:15,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-26 15:53:15,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-26 15:53:15,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-26 15:53:15,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-26 15:53:15,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-26 15:53:15,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-26 15:53:15,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-26 15:53:15,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-26 15:53:15,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-26 15:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-26 15:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-26 15:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-26 15:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-26 15:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-26 15:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-26 15:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-26 15:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-26 15:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-26 15:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-26 15:53:15,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-26 15:53:15,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-26 15:53:15,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-26 15:53:15,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-26 15:53:15,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-26 15:53:15,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-26 15:53:15,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-26 15:53:15,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-26 15:53:15,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-26 15:53:15,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-26 15:53:15,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-26 15:53:15,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-26 15:53:15,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-26 15:53:15,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-26 15:53:15,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-26 15:53:15,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-26 15:53:15,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-26 15:53:15,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-26 15:53:15,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-26 15:53:15,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-26 15:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-26 15:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-26 15:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-26 15:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-26 15:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-26 15:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-26 15:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-26 15:53:15,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-26 15:53:15,618 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:53:15,619 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-26 15:53:15,619 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-26 15:53:16,023 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 566 ms to try 5000 different iterations. Found a solution that moves 4693 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18126036670578866. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04031086338007644, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060615992452200326, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5214444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04031086338007644, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24533333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:53:16,039 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv181809038:81 , srv1105549297:82 , srv1243918645:82 , srv1291767190:82 , srv1923081264:82 , srv2061044220:82 , srv291204262:82 , srv590065965:82 , srv1088908861:83 , srv1194428226:83 , srv1272836954:83 , srv1688931384:83 , srv54728020:83 , srv682747403:83 , srv972460637:83 , srv1083132618:84 , srv1205105261:84 , srv153870199:84 , srv1782923150:84 , srv283639093:84 , srv477170496:84 , srv483834427:84 , srv654589811:84 , srv699527273:84 , srv722885846:84 , srv829234529:84 , srv918184816:84 , srv1014040807:85 , srv1181748615:85 , srv1231033875:85 , srv1253896178:85 , srv1503581201:85 , srv1536019568:85 , srv1710272573:85 , srv1895419594:85 , srv1943697046:85 , srv1970814084:85 , srv2117303428:85 , srv646016540:85 , srv692818989:85 , srv790062514:85 , srv860683481:85 , srv874021190:85 , srv1092094448:86 , srv1298541050:86 , srv1378381759:86 , srv1465220767:86 , srv178788037:86 , srv1879375683:86 , srv2017248143:86 , srv2019898724:86 , srv2113716144:86 , srv399340776:86 , srv556105946:86 , srv578592149:86 , srv656617629:86 , srv951791160:86 , srv1059234387:87 , srv1097281304:87 , srv1431258769:87 , srv1506312447:87 , srv1567763052:87 , srv1598949683:87 , srv2026219013:87 , srv2057735261:87 , srv341062472:87 , srv419230548:87 , srv1054912049:88 , srv1143979237:88 , srv1172173830:88 , srv1737156403:88 , srv1790325855:88 , srv1801315842:88 , srv1918221241:88 , srv2095005539:88 , srv375457675:88 , srv409739958:88 , srv460165825:88 , srv869475352:88 , srv1124706291:89 , srv1354676898:89 , srv144270541:89 , srv324600116:89 , srv37972116:89 , srv1670624263:90 , srv1723594481:90 , srv85235469:90 , srv997613201:90 , srv1399165214:91 , srv1575467824:91 , srv1956235166:91 , srv2058504092:91 , srv219533460:91 , srv1566945933:92 , srv1622400003:92 , srv1927038036:92 , srv1058948247:93 , srv1882056138:93 , srv880748095:96 , srv859917738:448 } 2023-11-26 15:53:16,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 15:53:16,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1172173830=11, srv1567763052=32, srv291204262=66, srv1895419594=48, srv1059234387=3, srv2113716144=62, srv1923081264=50, srv1181748615=12, srv2019898724=56, srv722885846=87, srv144270541=25, srv1058948247=2, srv874021190=94, srv219533460=64, srv1882056138=47, srv1927038036=51, srv1231033875=15, srv1737156403=40, srv181809038=45, srv2058504092=59, srv880748095=95, srv860683481=92, srv2117303428=63, srv1956235166=53, srv859917738=91, srv1088908861=5, srv1622400003=35, srv1723594481=39, srv1943697046=52, srv153870199=30, srv1205105261=14, srv1566945933=31, srv1536019568=29, srv1105549297=8, srv1431258769=24, srv1194428226=13, srv1575467824=33, srv1503581201=27, srv682747403=84, srv1083132618=4, srv869475352=93, srv1143979237=10, srv1670624263=36, srv1465220767=26, srv1272836954=18, srv283639093=65, srv1782923150=41, srv341062472=68, srv375457675=69, srv2095005539=61, srv460165825=74, srv829234529=89, srv409739958=72, srv918184816=96, srv1054912049=1, srv1124706291=9, srv178788037=42, srv1918221241=49, srv1378381759=22, srv37972116=70, srv590065965=80, srv656617629=83, srv1291767190=19, srv1506312447=28, srv1014040807=0, srv1879375683=46, srv1399165214=23, srv419230548=73, srv85235469=90, srv1092094448=6, srv1710272573=38, srv1688931384=37, srv477170496=75, srv1097281304=7, srv324600116=67, srv399340776=71, srv1253896178=17, srv1790325855=43, srv2026219013=57, srv699527273=86, srv2061044220=60, srv654589811=82, srv1243918645=16, srv692818989=85, srv790062514=88, srv1801315842=44, srv997613201=99, srv1354676898=21, srv1298541050=20, srv483834427=76, srv646016540=81, srv2057735261=58, srv54728020=77, srv578592149=79, srv972460637=98, srv556105946=78, srv1598949683=34, srv1970814084=54, srv951791160=97, srv2017248143=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 15:53:16,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 15:53:16,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 15:53:16,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 15:53:16,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 15:53:16,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 15:53:16,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 15:53:16,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 15:53:16,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 15:53:16,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 15:53:16,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 15:53:16,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 15:53:16,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 15:53:16,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 15:53:16,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 15:53:16,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 15:53:16,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 15:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 15:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 15:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 15:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 15:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 15:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 15:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 15:53:16,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 15:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 15:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 15:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 15:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 15:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 15:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 15:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-26 15:53:16,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-26 15:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-26 15:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-26 15:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-26 15:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-26 15:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-26 15:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-26 15:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-26 15:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-26 15:53:16,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-26 15:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-26 15:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-26 15:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-26 15:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-26 15:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-26 15:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-26 15:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-26 15:53:16,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-26 15:53:16,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-26 15:53:16,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-26 15:53:16,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-26 15:53:16,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-26 15:53:16,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-26 15:53:16,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-26 15:53:16,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-26 15:53:16,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-26 15:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-26 15:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-26 15:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-26 15:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-26 15:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-26 15:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-26 15:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-26 15:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-26 15:53:16,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-26 15:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-26 15:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-26 15:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-26 15:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-26 15:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-26 15:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-26 15:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-26 15:53:16,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-26 15:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-26 15:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-26 15:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-26 15:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-26 15:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-26 15:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-26 15:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-26 15:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-26 15:53:16,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-26 15:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-26 15:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-26 15:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-26 15:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-26 15:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-26 15:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-26 15:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-26 15:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-26 15:53:16,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-26 15:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-26 15:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-26 15:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-26 15:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-26 15:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-26 15:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-26 15:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-26 15:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-26 15:53:16,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-26 15:53:16,223 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04031086338007644, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060615992452200326, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04031086338007644, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24533333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:53:16,224 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-26 15:53:16,224 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1812275056174345, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04031086338007644, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060615992452200326, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04031086338007644, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24533333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-26 15:53:16,470 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 360 ms to try 5000 different iterations. Found a solution that moves 2650 regions; Going from a computed imbalance of 0.1812275056174345 to a new imbalance of 0.015607753143388178. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02227938775176151); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039363624699650684, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29444444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02227938775176151); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04833333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:53:16,484 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv880748095:76 , srv1058948247:77 , srv1566945933:79 , srv1882056138:79 , srv144270541:80 , srv2058504092:80 , srv1622400003:81 , srv1670624263:81 , srv1927038036:81 , srv219533460:81 , srv85235469:81 , srv997613201:81 , srv1399165214:82 , srv1737156403:82 , srv324600116:82 , srv37972116:82 , srv1054912049:83 , srv1124706291:83 , srv1354676898:83 , srv1575467824:83 , srv1801315842:83 , srv1956235166:83 , srv460165825:83 , srv1014040807:84 , srv1097281304:84 , srv1143979237:84 , srv1506312447:84 , srv1567763052:84 , srv1723594481:84 , srv1879375683:84 , srv2017248143:84 , srv2095005539:84 , srv419230548:84 , srv869475352:84 , srv1059234387:85 , srv1172173830:85 , srv1181748615:85 , srv1503581201:85 , srv178788037:85 , srv1790325855:85 , srv341062472:85 , srv409739958:85 , srv1194428226:86 , srv1378381759:86 , srv1465220767:86 , srv1536019568:86 , srv1918221241:86 , srv2026219013:86 , srv2117303428:86 , srv283639093:86 , srv646016540:86 , srv656617629:86 , srv829234529:86 , srv874021190:86 , srv951791160:86 , srv1083132618:87 , srv1092094448:87 , srv1205105261:87 , srv1253896178:87 , srv1298541050:87 , srv1431258769:87 , srv153870199:87 , srv1598949683:87 , srv1710272573:87 , srv1970814084:87 , srv375457675:87 , srv399340776:87 , srv477170496:87 , srv578592149:87 , srv790062514:87 , srv1105549297:88 , srv1272836954:88 , srv1688931384:88 , srv1782923150:88 , srv1895419594:88 , srv1943697046:88 , srv2019898724:88 , srv2057735261:88 , srv2061044220:88 , srv54728020:88 , srv556105946:88 , srv722885846:88 , srv918184816:88 , srv1231033875:89 , srv2113716144:89 , srv483834427:89 , srv590065965:89 , srv699527273:89 , srv860683481:89 , srv1088908861:90 , srv1291767190:90 , srv1923081264:90 , srv654589811:90 , srv682747403:90 , srv1243918645:91 , srv181809038:91 , srv291204262:91 , srv692818989:91 , srv972460637:91 , srv859917738:524 } 2023-11-26 15:53:16,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 15:53:16,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1172173830=11, srv1567763052=32, srv291204262=66, srv1895419594=48, srv1059234387=3, srv2113716144=62, srv1923081264=50, srv1181748615=12, srv2019898724=56, srv722885846=87, srv144270541=25, srv1058948247=2, srv874021190=94, srv219533460=64, srv1882056138=47, srv1927038036=51, srv1231033875=15, srv1737156403=40, srv181809038=45, srv2058504092=59, srv880748095=95, srv860683481=92, srv2117303428=63, srv1956235166=53, srv859917738=91, srv1088908861=5, srv1622400003=35, srv1723594481=39, srv1943697046=52, srv153870199=30, srv1205105261=14, srv1566945933=31, srv1536019568=29, srv1105549297=8, srv1431258769=24, srv1194428226=13, srv1575467824=33, srv1503581201=27, srv682747403=84, srv1083132618=4, srv869475352=93, srv1143979237=10, srv1670624263=36, srv1465220767=26, srv1272836954=18, srv283639093=65, srv1782923150=41, srv341062472=68, srv375457675=69, srv2095005539=61, srv460165825=74, srv829234529=89, srv409739958=72, srv918184816=96, srv1054912049=1, srv1124706291=9, srv178788037=42, srv1918221241=49, srv1378381759=22, srv37972116=70, srv590065965=80, srv656617629=83, srv1291767190=19, srv1506312447=28, srv1014040807=0, srv1879375683=46, srv1399165214=23, srv419230548=73, srv85235469=90, srv1092094448=6, srv1710272573=38, srv1688931384=37, srv477170496=75, srv1097281304=7, srv324600116=67, srv399340776=71, srv1253896178=17, srv1790325855=43, srv2026219013=57, srv699527273=86, srv2061044220=60, srv654589811=82, srv1243918645=16, srv692818989=85, srv790062514=88, srv1801315842=44, srv997613201=99, srv1354676898=21, srv1298541050=20, srv483834427=76, srv646016540=81, srv2057735261=58, srv54728020=77, srv578592149=79, srv972460637=98, srv556105946=78, srv1598949683=34, srv1970814084=54, srv951791160=97, srv2017248143=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 15:53:16,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 15:53:16,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 15:53:16,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 15:53:16,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 15:53:16,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 15:53:16,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 15:53:16,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 15:53:16,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 15:53:16,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 15:53:16,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 15:53:16,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 15:53:16,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 15:53:16,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 15:53:16,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 15:53:16,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 15:53:16,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 15:53:16,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 15:53:16,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 15:53:16,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 15:53:16,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 15:53:16,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 15:53:16,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 15:53:16,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 15:53:16,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 15:53:16,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 15:53:16,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 15:53:16,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 15:53:16,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 15:53:16,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 15:53:16,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 15:53:16,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-26 15:53:16,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-26 15:53:16,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-26 15:53:16,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-26 15:53:16,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-26 15:53:16,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-26 15:53:16,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-26 15:53:16,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-26 15:53:16,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-26 15:53:16,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-26 15:53:16,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-26 15:53:16,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-26 15:53:16,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-26 15:53:16,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-26 15:53:16,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-26 15:53:16,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-26 15:53:16,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-26 15:53:16,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-26 15:53:16,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-26 15:53:16,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-26 15:53:16,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-26 15:53:16,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-26 15:53:16,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-26 15:53:16,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-26 15:53:16,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-26 15:53:16,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-26 15:53:16,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-26 15:53:16,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-26 15:53:16,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-26 15:53:16,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-26 15:53:16,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-26 15:53:16,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-26 15:53:16,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-26 15:53:16,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-26 15:53:16,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-26 15:53:16,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-26 15:53:16,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-26 15:53:16,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-26 15:53:16,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-26 15:53:16,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-26 15:53:16,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-26 15:53:16,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-26 15:53:16,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-26 15:53:16,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-26 15:53:16,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-26 15:53:16,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-26 15:53:16,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-26 15:53:16,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-26 15:53:16,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-26 15:53:16,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-26 15:53:16,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-26 15:53:16,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-26 15:53:16,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-26 15:53:16,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-26 15:53:16,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-26 15:53:16,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-26 15:53:16,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-26 15:53:16,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-26 15:53:16,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-26 15:53:16,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-26 15:53:16,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-26 15:53:16,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-26 15:53:16,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-26 15:53:16,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-26 15:53:16,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-26 15:53:16,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-26 15:53:16,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-26 15:53:16,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-26 15:53:16,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-26 15:53:16,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-26 15:53:16,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-26 15:53:16,706 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02227938775176151); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039363624699650684, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02227938775176151); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04833333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:53:16,707 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-26 15:53:16,707 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015589197446789322, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02227938775176151); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039363624699650684, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02227938775176151); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04833333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-26 15:53:17,175 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 624 ms to try 5000 different iterations. Found a solution that moves 1001 regions; Going from a computed imbalance of 0.015589197446789322 to a new imbalance of 9.92027792199091E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007106690545187014); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01122167215373564); MoveCostFunction : (multiplier=7.0, imbalance=0.11122222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007106690545187014); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:53:17,181 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv692818989:81 , srv1231033875:82 , srv2057735261:82 , srv375457675:82 , srv1431258769:83 , srv1575467824:83 , srv1918221241:83 , srv2113716144:83 , srv556105946:83 , srv1088908861:84 , srv1092094448:84 , srv1143979237:84 , srv1172173830:84 , srv1399165214:84 , srv1598949683:84 , srv1622400003:84 , srv1723594481:84 , srv1790325855:84 , srv1882056138:84 , srv1923081264:84 , srv1927038036:84 , srv1943697046:84 , srv1956235166:84 , srv578592149:84 , srv654589811:84 , srv699527273:84 , srv790062514:84 , srv972460637:84 , srv1059234387:85 , srv1243918645:85 , srv1291767190:85 , srv1298541050:85 , srv1378381759:85 , srv1465220767:85 , srv1710272573:85 , srv181809038:85 , srv1895419594:85 , srv2019898724:85 , srv2026219013:85 , srv291204262:85 , srv341062472:85 , srv399340776:85 , srv409739958:85 , srv483834427:85 , srv682747403:85 , srv722885846:85 , srv860683481:85 , srv869475352:85 , srv951791160:85 , srv1054912049:86 , srv1097281304:86 , srv1124706291:86 , srv1253896178:86 , srv1354676898:86 , srv1506312447:86 , srv1536019568:86 , srv1566945933:86 , srv1567763052:86 , srv1688931384:86 , srv1782923150:86 , srv1801315842:86 , srv1970814084:86 , srv2058504092:86 , srv2095005539:86 , srv2117303428:86 , srv219533460:86 , srv283639093:86 , srv324600116:86 , srv419230548:86 , srv460165825:86 , srv477170496:86 , srv54728020:86 , srv590065965:86 , srv646016540:86 , srv656617629:86 , srv85235469:86 , srv880748095:86 , srv918184816:86 , srv1083132618:87 , srv1205105261:87 , srv1272836954:87 , srv1503581201:87 , srv153870199:87 , srv1670624263:87 , srv1737156403:87 , srv178788037:87 , srv1879375683:87 , srv2017248143:87 , srv2061044220:87 , srv37972116:87 , srv829234529:87 , srv874021190:87 , srv997613201:87 , srv1014040807:88 , srv1058948247:88 , srv1105549297:88 , srv1194428226:88 , srv144270541:88 , srv1181748615:89 , srv859917738:549 } 2023-11-26 15:53:17,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 15:53:17,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1172173830=11, srv1567763052=32, srv291204262=66, srv1895419594=48, srv1059234387=3, srv2113716144=62, srv1923081264=50, srv1181748615=12, srv2019898724=56, srv722885846=87, srv144270541=25, srv1058948247=2, srv874021190=94, srv219533460=64, srv1882056138=47, srv1927038036=51, srv1231033875=15, srv1737156403=40, srv181809038=45, srv2058504092=59, srv880748095=95, srv860683481=92, srv2117303428=63, srv1956235166=53, srv859917738=91, srv1088908861=5, srv1622400003=35, srv1723594481=39, srv1943697046=52, srv153870199=30, srv1205105261=14, srv1566945933=31, srv1536019568=29, srv1105549297=8, srv1431258769=24, srv1194428226=13, srv1575467824=33, srv1503581201=27, srv682747403=84, srv1083132618=4, srv869475352=93, srv1143979237=10, srv1670624263=36, srv1465220767=26, srv1272836954=18, srv283639093=65, srv1782923150=41, srv341062472=68, srv375457675=69, srv2095005539=61, srv460165825=74, srv829234529=89, srv409739958=72, srv918184816=96, srv1054912049=1, srv1124706291=9, srv178788037=42, srv1918221241=49, srv1378381759=22, srv37972116=70, srv590065965=80, srv656617629=83, srv1291767190=19, srv1506312447=28, srv1014040807=0, srv1879375683=46, srv1399165214=23, srv419230548=73, srv85235469=90, srv1092094448=6, srv1710272573=38, srv1688931384=37, srv477170496=75, srv1097281304=7, srv324600116=67, srv399340776=71, srv1253896178=17, srv1790325855=43, srv2026219013=57, srv699527273=86, srv2061044220=60, srv654589811=82, srv1243918645=16, srv692818989=85, srv790062514=88, srv1801315842=44, srv997613201=99, srv1354676898=21, srv1298541050=20, srv483834427=76, srv646016540=81, srv2057735261=58, srv54728020=77, srv578592149=79, srv972460637=98, srv556105946=78, srv1598949683=34, srv1970814084=54, srv951791160=97, srv2017248143=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 15:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 15:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 15:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 15:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 15:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 15:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 15:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 15:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 15:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 15:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 15:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 15:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 15:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 15:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 15:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 15:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 15:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-26 15:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-26 15:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-26 15:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-26 15:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-26 15:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-26 15:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-26 15:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-26 15:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-26 15:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-26 15:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-26 15:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-26 15:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-26 15:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-26 15:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-26 15:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-26 15:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-26 15:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-26 15:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-26 15:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-26 15:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-26 15:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-26 15:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-26 15:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-26 15:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-26 15:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-26 15:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-26 15:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-26 15:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-26 15:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-26 15:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-26 15:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-26 15:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-26 15:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-26 15:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-26 15:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-26 15:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-26 15:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-26 15:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-26 15:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-26 15:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-26 15:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-26 15:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-26 15:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-26 15:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-26 15:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-26 15:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-26 15:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-26 15:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-26 15:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-26 15:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-26 15:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-26 15:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-26 15:53:17,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-26 15:53:17,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-26 15:53:17,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-26 15:53:17,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-26 15:53:17,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-26 15:53:17,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-26 15:53:17,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-26 15:53:17,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-26 15:53:17,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-26 15:53:17,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-26 15:53:17,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-26 15:53:17,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-26 15:53:17,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-26 15:53:17,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-26 15:53:17,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-26 15:53:17,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-26 15:53:17,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-26 15:53:17,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-26 15:53:17,326 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007106690545187014); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01122167215373564); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007106690545187014); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:53:17,326 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-26 15:53:17,326 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.850186403894854E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007106690545187014); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01122167215373564); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007106690545187014); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-26 15:53:17,696 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 476 ms to try 5000 different iterations. Found a solution that moves 304 regions; Going from a computed imbalance of 9.850186403894854E-4 to a new imbalance of 2.8435621527596718E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.317058262388458E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.033777777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.317058262388458E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:53:17,697 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1181748615:83 , srv2019898724:83 , srv219533460:83 , srv399340776:83 , srv1058948247:84 , srv1059234387:84 , srv1083132618:84 , srv1097281304:84 , srv1105549297:84 , srv1124706291:84 , srv1205105261:84 , srv1243918645:84 , srv1253896178:84 , srv1272836954:84 , srv1291767190:84 , srv1298541050:84 , srv1399165214:84 , srv153870199:84 , srv1598949683:84 , srv1622400003:84 , srv1670624263:84 , srv1723594481:84 , srv1782923150:84 , srv178788037:84 , srv1895419594:84 , srv1927038036:84 , srv1956235166:84 , srv2017248143:84 , srv2026219013:84 , srv2113716144:84 , srv2117303428:84 , srv291204262:84 , srv324600116:84 , srv37972116:84 , srv656617629:84 , srv722885846:84 , srv860683481:84 , srv874021190:84 , srv880748095:84 , srv918184816:84 , srv972460637:84 , srv997613201:84 , srv1014040807:85 , srv1054912049:85 , srv1088908861:85 , srv1092094448:85 , srv1172173830:85 , srv1194428226:85 , srv1231033875:85 , srv1354676898:85 , srv1378381759:85 , srv144270541:85 , srv1465220767:85 , srv1503581201:85 , srv1506312447:85 , srv1536019568:85 , srv1566945933:85 , srv1567763052:85 , srv1737156403:85 , srv1790325855:85 , srv181809038:85 , srv1879375683:85 , srv1882056138:85 , srv1918221241:85 , srv1923081264:85 , srv1970814084:85 , srv2057735261:85 , srv2058504092:85 , srv2061044220:85 , srv2095005539:85 , srv341062472:85 , srv375457675:85 , srv409739958:85 , srv419230548:85 , srv477170496:85 , srv483834427:85 , srv54728020:85 , srv556105946:85 , srv578592149:85 , srv646016540:85 , srv654589811:85 , srv682747403:85 , srv692818989:85 , srv699527273:85 , srv790062514:85 , srv829234529:85 , srv85235469:85 , srv869475352:85 , srv951791160:85 , srv1143979237:86 , srv1431258769:86 , srv1575467824:86 , srv1688931384:86 , srv1710272573:86 , srv1801315842:86 , srv1943697046:86 , srv283639093:86 , srv460165825:86 , srv590065965:86 , srv859917738:621 } 2023-11-26 15:53:17,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 15:53:17,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1172173830=11, srv1567763052=32, srv291204262=66, srv1895419594=48, srv1059234387=3, srv2113716144=62, srv1923081264=50, srv1181748615=12, srv2019898724=56, srv722885846=87, srv144270541=25, srv1058948247=2, srv874021190=94, srv219533460=64, srv1882056138=47, srv1927038036=51, srv1231033875=15, srv1737156403=40, srv181809038=45, srv2058504092=59, srv880748095=95, srv860683481=92, srv2117303428=63, srv1956235166=53, srv859917738=91, srv1088908861=5, srv1622400003=35, srv1723594481=39, srv1943697046=52, srv153870199=30, srv1205105261=14, srv1566945933=31, srv1536019568=29, srv1105549297=8, srv1431258769=24, srv1194428226=13, srv1575467824=33, srv1503581201=27, srv682747403=84, srv1083132618=4, srv869475352=93, srv1143979237=10, srv1670624263=36, srv1465220767=26, srv1272836954=18, srv283639093=65, srv1782923150=41, srv341062472=68, srv375457675=69, srv2095005539=61, srv460165825=74, srv829234529=89, srv409739958=72, srv918184816=96, srv1054912049=1, srv1124706291=9, srv178788037=42, srv1918221241=49, srv1378381759=22, srv37972116=70, srv590065965=80, srv656617629=83, srv1291767190=19, srv1506312447=28, srv1014040807=0, srv1879375683=46, srv1399165214=23, srv419230548=73, srv85235469=90, srv1092094448=6, srv1710272573=38, srv1688931384=37, srv477170496=75, srv1097281304=7, srv324600116=67, srv399340776=71, srv1253896178=17, srv1790325855=43, srv2026219013=57, srv699527273=86, srv2061044220=60, srv654589811=82, srv1243918645=16, srv692818989=85, srv790062514=88, srv1801315842=44, srv997613201=99, srv1354676898=21, srv1298541050=20, srv483834427=76, srv646016540=81, srv2057735261=58, srv54728020=77, srv578592149=79, srv972460637=98, srv556105946=78, srv1598949683=34, srv1970814084=54, srv951791160=97, srv2017248143=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 15:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 15:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 15:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 15:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 15:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 15:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 15:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 15:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 15:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 15:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 15:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-26 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-26 15:53:17,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-26 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-26 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-26 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-26 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-26 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-26 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-26 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-26 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-26 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-26 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-26 15:53:17,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-26 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-26 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-26 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-26 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-26 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-26 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-26 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-26 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-26 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-26 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-26 15:53:17,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-26 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-26 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-26 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-26 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-26 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-26 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-26 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-26 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-26 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-26 15:53:17,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-26 15:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-26 15:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-26 15:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-26 15:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-26 15:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-26 15:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-26 15:53:17,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-26 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-26 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-26 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-26 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-26 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-26 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-26 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-26 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-26 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-26 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-26 15:53:17,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-26 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-26 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-26 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-26 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-26 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-26 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-26 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-26 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-26 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-26 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-26 15:53:17,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-26 15:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-26 15:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-26 15:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-26 15:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-26 15:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-26 15:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-26 15:53:17,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-26 15:53:17,881 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.317058262388458E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.317058262388458E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:53:17,881 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-26 15:53:17,881 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6306968030973254E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=6.317058262388458E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=6.317058262388458E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-26 15:53:18,177 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 428 ms to try 5000 different iterations. Found a solution that moves 28 regions; Going from a computed imbalance of 2.6306968030973254E-5 to a new imbalance of 7.583817690791035E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.003111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:53:18,177 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1354676898:83 , srv1431258769:83 , srv1575467824:83 , srv1688931384:83 , srv1710272573:83 , srv1801315842:83 , srv1943697046:83 , srv1970814084:83 , srv2095005539:83 , srv409739958:83 , srv460165825:83 , srv483834427:83 , srv590065965:83 , srv654589811:83 , srv682747403:83 , srv860683481:83 , srv1014040807:84 , srv1054912049:84 , srv1058948247:84 , srv1083132618:84 , srv1088908861:84 , srv1092094448:84 , srv1105549297:84 , srv1124706291:84 , srv1143979237:84 , srv1172173830:84 , srv1181748615:84 , srv1194428226:84 , srv1205105261:84 , srv1243918645:84 , srv1253896178:84 , srv1272836954:84 , srv1298541050:84 , srv1378381759:84 , srv144270541:84 , srv1465220767:84 , srv1503581201:84 , srv1506312447:84 , srv1536019568:84 , srv153870199:84 , srv1566945933:84 , srv1567763052:84 , srv1598949683:84 , srv1670624263:84 , srv1723594481:84 , srv1737156403:84 , srv1782923150:84 , srv178788037:84 , srv1790325855:84 , srv181809038:84 , srv1879375683:84 , srv1895419594:84 , srv1918221241:84 , srv1956235166:84 , srv2019898724:84 , srv2026219013:84 , srv2057735261:84 , srv2058504092:84 , srv2061044220:84 , srv2113716144:84 , srv283639093:84 , srv291204262:84 , srv341062472:84 , srv375457675:84 , srv37972116:84 , srv419230548:84 , srv477170496:84 , srv54728020:84 , srv556105946:84 , srv578592149:84 , srv646016540:84 , srv656617629:84 , srv692818989:84 , srv699527273:84 , srv829234529:84 , srv85235469:84 , srv869475352:84 , srv874021190:84 , srv880748095:84 , srv918184816:84 , srv951791160:84 , srv972460637:84 , srv997613201:84 , srv1059234387:85 , srv1097281304:85 , srv1231033875:85 , srv1291767190:85 , srv1399165214:85 , srv1622400003:85 , srv1882056138:85 , srv1923081264:85 , srv1927038036:85 , srv2017248143:85 , srv2117303428:85 , srv219533460:85 , srv324600116:85 , srv399340776:85 , srv722885846:85 , srv790062514:85 , srv859917738:684 } 2023-11-26 15:53:18,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 15:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1172173830=11, srv1567763052=32, srv291204262=66, srv1895419594=48, srv1059234387=3, srv2113716144=62, srv1923081264=50, srv1181748615=12, srv2019898724=56, srv722885846=87, srv144270541=25, srv1058948247=2, srv874021190=94, srv219533460=64, srv1882056138=47, srv1927038036=51, srv1231033875=15, srv1737156403=40, srv181809038=45, srv2058504092=59, srv880748095=95, srv860683481=92, srv2117303428=63, srv1956235166=53, srv859917738=91, srv1088908861=5, srv1622400003=35, srv1723594481=39, srv1943697046=52, srv153870199=30, srv1205105261=14, srv1566945933=31, srv1536019568=29, srv1105549297=8, srv1431258769=24, srv1194428226=13, srv1575467824=33, srv1503581201=27, srv682747403=84, srv1083132618=4, srv869475352=93, srv1143979237=10, srv1670624263=36, srv1465220767=26, srv1272836954=18, srv283639093=65, srv1782923150=41, srv341062472=68, srv375457675=69, srv2095005539=61, srv460165825=74, srv829234529=89, srv409739958=72, srv918184816=96, srv1054912049=1, srv1124706291=9, srv178788037=42, srv1918221241=49, srv1378381759=22, srv37972116=70, srv590065965=80, srv656617629=83, srv1291767190=19, srv1506312447=28, srv1014040807=0, srv1879375683=46, srv1399165214=23, srv419230548=73, srv85235469=90, srv1092094448=6, srv1710272573=38, srv1688931384=37, srv477170496=75, srv1097281304=7, srv324600116=67, srv399340776=71, srv1253896178=17, srv1790325855=43, srv2026219013=57, srv699527273=86, srv2061044220=60, srv654589811=82, srv1243918645=16, srv692818989=85, srv790062514=88, srv1801315842=44, srv997613201=99, srv1354676898=21, srv1298541050=20, srv483834427=76, srv646016540=81, srv2057735261=58, srv54728020=77, srv578592149=79, srv972460637=98, srv556105946=78, srv1598949683=34, srv1970814084=54, srv951791160=97, srv2017248143=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 15:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 15:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 15:53:18,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 15:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 15:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 15:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 15:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 15:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 15:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 15:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 15:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 15:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 15:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 15:53:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 15:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 15:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 15:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 15:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 15:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 15:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 15:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 15:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 15:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 15:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 15:53:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 15:53:18,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 15:53:18,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 15:53:18,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 15:53:18,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 15:53:18,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 15:53:18,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-26 15:53:18,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-26 15:53:18,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-26 15:53:18,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-26 15:53:18,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-26 15:53:18,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-26 15:53:18,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-26 15:53:18,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-26 15:53:18,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-26 15:53:18,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-26 15:53:18,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-26 15:53:18,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-26 15:53:18,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-26 15:53:18,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-26 15:53:18,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-26 15:53:18,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-26 15:53:18,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-26 15:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-26 15:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-26 15:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-26 15:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-26 15:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-26 15:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-26 15:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-26 15:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-26 15:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-26 15:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-26 15:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-26 15:53:18,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-26 15:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-26 15:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-26 15:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-26 15:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-26 15:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-26 15:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-26 15:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-26 15:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-26 15:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-26 15:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-26 15:53:18,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-26 15:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-26 15:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-26 15:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-26 15:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-26 15:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-26 15:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-26 15:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-26 15:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-26 15:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-26 15:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-26 15:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-26 15:53:18,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-26 15:53:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-26 15:53:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-26 15:53:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-26 15:53:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-26 15:53:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-26 15:53:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-26 15:53:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-26 15:53:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-26 15:53:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-26 15:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-26 15:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-26 15:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-26 15:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-26 15:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-26 15:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-26 15:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-26 15:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-26 15:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-26 15:53:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-26 15:53:18,323 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:53:18,324 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1354676898:83 , srv1431258769:83 , srv1575467824:83 , srv1688931384:83 , srv1710272573:83 , srv1801315842:83 , srv1943697046:83 , srv1970814084:83 , srv2095005539:83 , srv409739958:83 , srv460165825:83 , srv483834427:83 , srv590065965:83 , srv654589811:83 , srv682747403:83 , srv860683481:83 , srv1014040807:84 , srv1054912049:84 , srv1058948247:84 , srv1083132618:84 , srv1088908861:84 , srv1092094448:84 , srv1105549297:84 , srv1124706291:84 , srv1143979237:84 , srv1172173830:84 , srv1181748615:84 , srv1194428226:84 , srv1205105261:84 , srv1243918645:84 , srv1253896178:84 , srv1272836954:84 , srv1298541050:84 , srv1378381759:84 , srv144270541:84 , srv1465220767:84 , srv1503581201:84 , srv1506312447:84 , srv1536019568:84 , srv153870199:84 , srv1566945933:84 , srv1567763052:84 , srv1598949683:84 , srv1670624263:84 , srv1723594481:84 , srv1737156403:84 , srv1782923150:84 , srv178788037:84 , srv1790325855:84 , srv181809038:84 , srv1879375683:84 , srv1895419594:84 , srv1918221241:84 , srv1956235166:84 , srv2019898724:84 , srv2026219013:84 , srv2057735261:84 , srv2058504092:84 , srv2061044220:84 , srv2113716144:84 , srv283639093:84 , srv291204262:84 , srv341062472:84 , srv375457675:84 , srv37972116:84 , srv419230548:84 , srv477170496:84 , srv54728020:84 , srv556105946:84 , srv578592149:84 , srv646016540:84 , srv656617629:84 , srv692818989:84 , srv699527273:84 , srv829234529:84 , srv85235469:84 , srv869475352:84 , srv874021190:84 , srv880748095:84 , srv918184816:84 , srv951791160:84 , srv972460637:84 , srv997613201:84 , srv1059234387:85 , srv1097281304:85 , srv1231033875:85 , srv1291767190:85 , srv1399165214:85 , srv1622400003:85 , srv1882056138:85 , srv1923081264:85 , srv1927038036:85 , srv2017248143:85 , srv2117303428:85 , srv219533460:85 , srv324600116:85 , srv399340776:85 , srv722885846:85 , srv790062514:85 , srv859917738:684 } 2023-11-26 15:53:18,357 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=401 (was 401), ProcessCount=172 (was 172), AvailableMemoryMB=8828 (was 9748) 2023-11-26 15:53:18,367 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=401, ProcessCount=172, AvailableMemoryMB=8824 2023-11-26 15:53:18,367 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-26 15:53:18,367 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-11-26 15:53:18,380 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1061879214:84 , srv1101640128:84 , srv1121032814:84 , srv1157124104:84 , srv1289094916:84 , srv1350391007:84 , srv1432521594:84 , srv1497338627:84 , srv1548947493:84 , srv1722468290:84 , srv1738874827:84 , srv1787602038:84 , srv1810752448:84 , srv1817227488:84 , srv1928456694:84 , srv2009334606:84 , srv2143024850:84 , srv302017648:84 , srv315203200:84 , srv381187525:84 , srv468575216:84 , srv590254181:84 , srv596750223:84 , srv599633667:84 , srv642893812:84 , srv700955357:84 , srv759371643:84 , srv805267139:84 , srv940362891:84 , srv932359312:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-11-26 15:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 15:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1548947493=8, srv599633667=23, srv1817227488=13, srv1928456694=14, srv302017648=17, srv932359312=28, srv1738874827=10, srv1787602038=11, srv642893812=24, srv1061879214=0, srv1157124104=3, srv468575216=20, srv1722468290=9, srv805267139=27, srv1350391007=5, srv381187525=19, srv2009334606=15, srv700955357=25, srv590254181=21, srv940362891=29, srv596750223=22, srv1121032814=2, srv1432521594=6, srv1810752448=12, srv315203200=18, srv1101640128=1, srv759371643=26, srv2143024850=16, srv1289094916=4, srv1497338627=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 15:53:18,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 15:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 15:53:18,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 15:53:18,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 15:53:18,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 15:53:18,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 15:53:18,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 15:53:18,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 15:53:18,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-11-26 15:53:18,415 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1059149982594936, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:53:18,416 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912526777965354, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1059149982594936, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-11-26 15:55:18,399 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5715615 different iterations. Found a solution that moves 1853 regions; Going from a computed imbalance of 0.9912526777965354 to a new imbalance of 4.324994439959734E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6862962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:55:18,406 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1061879214:90 , srv1101640128:90 , srv1121032814:90 , srv1157124104:90 , srv1289094916:90 , srv1350391007:90 , srv1432521594:90 , srv1497338627:90 , srv1548947493:90 , srv1722468290:90 , srv1738874827:90 , srv1787602038:90 , srv1810752448:90 , srv1817227488:90 , srv1928456694:90 , srv2009334606:90 , srv2143024850:90 , srv302017648:90 , srv315203200:90 , srv381187525:90 , srv468575216:90 , srv590254181:90 , srv596750223:90 , srv599633667:90 , srv642893812:90 , srv700955357:90 , srv759371643:90 , srv805267139:90 , srv932359312:90 , srv940362891:90 } 2023-11-26 15:55:18,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-26 15:55:18,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1548947493=8, srv599633667=23, srv1817227488=13, srv1928456694=14, srv302017648=17, srv932359312=28, srv1738874827=10, srv1787602038=11, srv642893812=24, srv1061879214=0, srv1157124104=3, srv468575216=20, srv1722468290=9, srv805267139=27, srv1350391007=5, srv381187525=19, srv2009334606=15, srv700955357=25, srv590254181=21, srv940362891=29, srv596750223=22, srv1121032814=2, srv1432521594=6, srv1810752448=12, srv315203200=18, srv1101640128=1, srv759371643=26, srv2143024850=16, srv1289094916=4, srv1497338627=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-26 15:55:18,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-26 15:55:18,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-26 15:55:18,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-26 15:55:18,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-26 15:55:18,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-26 15:55:18,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-26 15:55:18,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-26 15:55:18,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-26 15:55:18,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-26 15:55:18,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-26 15:55:18,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-26 15:55:18,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-26 15:55:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-26 15:55:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-26 15:55:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-26 15:55:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-26 15:55:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-26 15:55:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-26 15:55:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-26 15:55:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-26 15:55:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-26 15:55:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-26 15:55:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-26 15:55:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-26 15:55:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-26 15:55:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-26 15:55:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-26 15:55:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-26 15:55:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-26 15:55:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-26 15:55:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-11-26 15:55:18,433 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-26 15:55:18,433 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1061879214:90 , srv1101640128:90 , srv1121032814:90 , srv1157124104:90 , srv1289094916:90 , srv1350391007:90 , srv1432521594:90 , srv1497338627:90 , srv1548947493:90 , srv1722468290:90 , srv1738874827:90 , srv1787602038:90 , srv1810752448:90 , srv1817227488:90 , srv1928456694:90 , srv2009334606:90 , srv2143024850:90 , srv302017648:90 , srv315203200:90 , srv381187525:90 , srv468575216:90 , srv590254181:90 , srv596750223:90 , srv599633667:90 , srv642893812:90 , srv700955357:90 , srv759371643:90 , srv805267139:90 , srv932359312:90 , srv940362891:90 } 2023-11-26 15:55:18,453 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=198 (was 401), ProcessCount=169 (was 172), AvailableMemoryMB=8273 (was 8824)