2023-12-02 02:02:32,245 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-02 02:02:32,489 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-02 02:02:32,587 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-02 02:02:32,641 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-02 02:02:32,647 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-02 02:02:32,657 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=501, ProcessCount=166, AvailableMemoryMB=6461 2023-12-02 02:02:32,659 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-02 02:02:32,659 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-02 02:02:32,934 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1019032799:84 , srv1019705160:84 , srv1029472809:84 , srv1057555842:84 , srv1074664479:84 , srv1120213467:84 , srv115168499:84 , srv1161210530:84 , srv1182994292:84 , srv1188270837:84 , srv1218384490:84 , srv1219213250:84 , srv1261294239:84 , srv127913181:84 , srv1310522606:84 , srv1320966798:84 , srv1354733259:84 , srv1379277637:84 , srv1383195097:84 , srv1467043269:84 , srv1482124747:84 , srv1483293953:84 , srv14908826:84 , srv1504767708:84 , srv1508624499:84 , srv1525104389:84 , srv153574395:84 , srv1539463812:84 , srv1569115301:84 , srv1570460137:84 , srv1597344633:84 , srv1622867008:84 , srv1624272157:84 , srv1690760344:84 , srv1692305515:84 , srv1697378383:84 , srv1740077706:84 , srv1740290398:84 , srv1744057218:84 , srv1748222084:84 , srv175381939:84 , srv1784705839:84 , srv1792629538:84 , srv1810279343:84 , srv1833571904:84 , srv1843373500:84 , srv1846569281:84 , srv1851902294:84 , srv1858165255:84 , srv1867433268:84 , srv1891961830:84 , srv194447655:84 , srv1958642984:84 , srv1987086198:84 , srv1993428690:84 , srv2016747821:84 , srv2072700207:84 , srv2075500790:84 , srv2086075993:84 , srv2091282872:84 , srv2103672311:84 , srv216949931:84 , srv230652135:84 , srv233797076:84 , srv24468510:84 , srv246236393:84 , srv252367900:84 , srv258153003:84 , srv262247760:84 , srv265914539:84 , srv369883873:84 , srv379136620:84 , srv398220342:84 , srv412601433:84 , srv43154857:84 , srv442535629:84 , srv451696727:84 , srv464426028:84 , srv474865244:84 , srv476325939:84 , srv478096288:84 , srv47918257:84 , srv497648006:84 , srv625031552:84 , srv659962339:84 , srv673251282:84 , srv703565032:84 , srv714531569:84 , srv733149896:84 , srv7725176:84 , srv821141:84 , srv826825372:84 , srv854238996:84 , srv882600871:84 , srv892172166:84 , srv921022688:84 , srv940989308:84 , srv943395902:84 , srv999971952:84 , srv861246482:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-02 02:02:32,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-02 02:02:32,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv369883873=70, srv194447655=51, srv625031552=83, srv1624272157=32, srv854238996=92, srv1597344633=30, srv1188270837=9, srv153574395=26, srv246236393=65, srv1261294239=12, srv442535629=75, srv1019705160=1, srv673251282=85, srv2016747821=55, srv733149896=88, srv1504767708=23, srv1891961830=50, srv1525104389=25, srv1379277637=17, srv47918257=81, srv262247760=68, srv826825372=91, srv1074664479=4, srv1697378383=35, srv1744057218=38, srv2091282872=59, srv398220342=72, srv1784705839=41, srv1810279343=43, srv1987086198=53, srv175381939=40, srv1833571904=44, srv1219213250=11, srv1383195097=18, srv474865244=78, srv1310522606=14, srv861246482=93, srv1993428690=54, srv1182994292=8, srv265914539=69, srv1690760344=33, srv1622867008=31, srv1740077706=36, srv2086075993=58, srv1019032799=0, srv216949931=61, srv1483293953=21, srv464426028=77, srv14908826=22, srv1846569281=46, srv497648006=82, srv1858165255=48, srv1029472809=2, srv233797076=63, srv1120213467=5, srv1161210530=7, srv2075500790=57, srv1539463812=27, srv2072700207=56, srv1508624499=24, srv476325939=79, srv892172166=95, srv127913181=13, srv1740290398=37, srv7725176=89, srv24468510=64, srv999971952=99, srv882600871=94, srv412601433=73, srv252367900=66, srv1748222084=39, srv1792629538=42, srv451696727=76, srv1057555842=3, srv2103672311=60, srv43154857=74, srv258153003=67, srv1482124747=20, srv1867433268=49, srv1958642984=52, srv1218384490=10, srv821141=90, srv940989308=97, srv1843373500=45, srv478096288=80, srv230652135=62, srv379136620=71, srv1569115301=28, srv1320966798=15, srv1354733259=16, srv659962339=84, srv1570460137=29, srv1692305515=34, srv1467043269=19, srv714531569=87, srv921022688=96, srv115168499=6, srv1851902294=47, srv703565032=86, srv943395902=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-02 02:02:33,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-02 02:02:33,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-02 02:02:33,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-02 02:02:33,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-02 02:02:33,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-02 02:02:33,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-02 02:02:33,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-02 02:02:33,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-02 02:02:33,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-02 02:02:33,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-02 02:02:33,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-02 02:02:33,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-02 02:02:33,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-02 02:02:33,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-02 02:02:33,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-02 02:02:33,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-02 02:02:33,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-02 02:02:33,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-02 02:02:33,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-02 02:02:33,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-02 02:02:33,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-02 02:02:33,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-02 02:02:33,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-02 02:02:33,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-02 02:02:33,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-02 02:02:33,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-02 02:02:33,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-02 02:02:33,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-02 02:02:33,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-02 02:02:33,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-02 02:02:33,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-02 02:02:33,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-02 02:02:33,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-02 02:02:33,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-02 02:02:33,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-02 02:02:33,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-02 02:02:33,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-02 02:02:33,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-02 02:02:33,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-02 02:02:33,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-02 02:02:33,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-02 02:02:33,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-02 02:02:33,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-02 02:02:33,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-02 02:02:33,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-02 02:02:33,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-02 02:02:33,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-02 02:02:33,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-02 02:02:33,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-02 02:02:33,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-02 02:02:33,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-02 02:02:33,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-02 02:02:33,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-02 02:02:33,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-02 02:02:33,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-02 02:02:33,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-02 02:02:33,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-02 02:02:33,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-02 02:02:33,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-02 02:02:33,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-02 02:02:33,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-02 02:02:33,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-02 02:02:33,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-02 02:02:33,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-02 02:02:33,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-02 02:02:33,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-02 02:02:33,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-02 02:02:33,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-02 02:02:33,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-02 02:02:33,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-02 02:02:33,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-02 02:02:33,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-02 02:02:33,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-02 02:02:33,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-02 02:02:33,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-02 02:02:33,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-02 02:02:33,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-02 02:02:33,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-02 02:02:33,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-02 02:02:33,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-02 02:02:33,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-02 02:02:33,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-02 02:02:33,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-02 02:02:33,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-02 02:02:33,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-02 02:02:33,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-02 02:02:33,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-02 02:02:33,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-02 02:02:33,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-02 02:02:33,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-02 02:02:33,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-02 02:02:33,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-02 02:02:33,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-02 02:02:33,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-02 02:02:33,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-02 02:02:33,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-02 02:02:33,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-02 02:02:33,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-02 02:02:33,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-02 02:02:33,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-02 02:02:33,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-02 02:02:33,291 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:02:33,292 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-02 02:02:33,292 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-02 02:02:33,765 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 678 ms to try 5000 different iterations. Found a solution that moves 4715 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17781378371021767. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04172457629342918, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06006356528127964, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5238888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04172457629342918, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1725, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:02:33,773 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1019705160:82 , srv115168499:82 , srv24468510:82 , srv258153003:82 , srv892172166:82 , srv943395902:82 , srv1482124747:83 , srv1569115301:83 , srv1740077706:83 , srv1792629538:83 , srv2103672311:83 , srv476325939:83 , srv673251282:83 , srv1019032799:84 , srv1310522606:84 , srv1504767708:84 , srv1993428690:84 , srv252367900:84 , srv265914539:84 , srv821141:84 , srv854238996:84 , srv1120213467:85 , srv1483293953:85 , srv1525104389:85 , srv1622867008:85 , srv1624272157:85 , srv1744057218:85 , srv1846569281:85 , srv1858165255:85 , srv1867433268:85 , srv2075500790:85 , srv230652135:85 , srv233797076:85 , srv262247760:85 , srv398220342:85 , srv474865244:85 , srv703565032:85 , srv733149896:85 , srv882600871:85 , srv1161210530:86 , srv1188270837:86 , srv1570460137:86 , srv1597344633:86 , srv1692305515:86 , srv1697378383:86 , srv175381939:86 , srv1843373500:86 , srv194447655:86 , srv2072700207:86 , srv2091282872:86 , srv246236393:86 , srv369883873:86 , srv412601433:86 , srv47918257:86 , srv999971952:86 , srv1074664479:87 , srv1182994292:87 , srv1218384490:87 , srv1354733259:87 , srv14908826:87 , srv1508624499:87 , srv1833571904:87 , srv1958642984:87 , srv1987086198:87 , srv2086075993:87 , srv464426028:87 , srv497648006:87 , srv659962339:87 , srv714531569:87 , srv826825372:87 , srv940989308:87 , srv1057555842:88 , srv1261294239:88 , srv127913181:88 , srv1320966798:88 , srv1383195097:88 , srv1467043269:88 , srv1740290398:88 , srv1784705839:88 , srv1810279343:88 , srv216949931:88 , srv43154857:88 , srv451696727:88 , srv478096288:88 , srv625031552:88 , srv7725176:88 , srv1029472809:89 , srv1690760344:89 , srv1748222084:89 , srv1891961830:89 , srv921022688:89 , srv153574395:90 , srv1851902294:90 , srv379136620:90 , srv442535629:90 , srv1539463812:91 , srv1219213250:93 , srv1379277637:93 , srv2016747821:94 , srv861246482:461 } 2023-12-02 02:02:33,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-02 02:02:33,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv369883873=70, srv194447655=51, srv625031552=83, srv1624272157=32, srv854238996=92, srv1597344633=30, srv1188270837=9, srv153574395=26, srv246236393=65, srv1261294239=12, srv442535629=75, srv1019705160=1, srv673251282=85, srv2016747821=55, srv733149896=88, srv1504767708=23, srv1891961830=50, srv1525104389=25, srv1379277637=17, srv47918257=81, srv262247760=68, srv826825372=91, srv1074664479=4, srv1697378383=35, srv1744057218=38, srv2091282872=59, srv398220342=72, srv1784705839=41, srv1810279343=43, srv1987086198=53, srv175381939=40, srv1833571904=44, srv1219213250=11, srv1383195097=18, srv474865244=78, srv1310522606=14, srv861246482=93, srv1993428690=54, srv1182994292=8, srv265914539=69, srv1690760344=33, srv1622867008=31, srv1740077706=36, srv2086075993=58, srv1019032799=0, srv216949931=61, srv1483293953=21, srv464426028=77, srv14908826=22, srv1846569281=46, srv497648006=82, srv1858165255=48, srv1029472809=2, srv233797076=63, srv1120213467=5, srv1161210530=7, srv2075500790=57, srv1539463812=27, srv2072700207=56, srv1508624499=24, srv476325939=79, srv892172166=95, srv127913181=13, srv1740290398=37, srv7725176=89, srv24468510=64, srv999971952=99, srv882600871=94, srv412601433=73, srv252367900=66, srv1748222084=39, srv1792629538=42, srv451696727=76, srv1057555842=3, srv2103672311=60, srv43154857=74, srv258153003=67, srv1482124747=20, srv1867433268=49, srv1958642984=52, srv1218384490=10, srv821141=90, srv940989308=97, srv1843373500=45, srv478096288=80, srv230652135=62, srv379136620=71, srv1569115301=28, srv1320966798=15, srv1354733259=16, srv659962339=84, srv1570460137=29, srv1692305515=34, srv1467043269=19, srv714531569=87, srv921022688=96, srv115168499=6, srv1851902294=47, srv703565032=86, srv943395902=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-02 02:02:33,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-02 02:02:33,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-02 02:02:33,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-02 02:02:33,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-02 02:02:33,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-02 02:02:33,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-02 02:02:33,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-02 02:02:33,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-02 02:02:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-02 02:02:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-02 02:02:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-02 02:02:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-02 02:02:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-02 02:02:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-02 02:02:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-02 02:02:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-02 02:02:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-02 02:02:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-02 02:02:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-02 02:02:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-02 02:02:33,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-02 02:02:33,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-02 02:02:33,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-02 02:02:33,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-02 02:02:33,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-02 02:02:33,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-02 02:02:33,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-02 02:02:33,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-02 02:02:33,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-02 02:02:33,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-02 02:02:33,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-02 02:02:33,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-02 02:02:33,890 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04172457629342918, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06006356528127964, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04172457629342918, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1725, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:02:33,890 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-02 02:02:33,891 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.177780768574571, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04172457629342918, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06006356528127964, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04172457629342918, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1725, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-02 02:02:34,070 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 256 ms to try 5000 different iterations. Found a solution that moves 2633 regions; Going from a computed imbalance of 0.177780768574571 to a new imbalance of 0.014488655398773333. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02541291377541991, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03909182260828038, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29255555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02541291377541991, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:02:34,079 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2016747821:78 , srv1219213250:80 , srv1379277637:81 , srv1748222084:81 , srv1851902294:81 , srv43154857:81 , srv442535629:81 , srv1539463812:82 , srv1690760344:82 , srv379136620:82 , srv921022688:82 , srv1074664479:83 , srv1810279343:83 , srv1891961830:83 , srv216949931:83 , srv412601433:83 , srv464426028:83 , srv478096288:83 , srv940989308:83 , srv1029472809:84 , srv1057555842:84 , srv1320966798:84 , srv1383195097:84 , srv1467043269:84 , srv1508624499:84 , srv1692305515:84 , srv1740290398:84 , srv175381939:84 , srv1987086198:84 , srv2075500790:84 , srv2086075993:84 , srv497648006:84 , srv625031552:84 , srv659962339:84 , srv7725176:84 , srv1261294239:85 , srv1354733259:85 , srv1483293953:85 , srv1525104389:85 , srv153574395:85 , srv1597344633:85 , srv1744057218:85 , srv1784705839:85 , srv1833571904:85 , srv1843373500:85 , srv1958642984:85 , srv2072700207:85 , srv233797076:85 , srv246236393:85 , srv262247760:85 , srv369883873:85 , srv451696727:85 , srv882600871:85 , srv999971952:85 , srv1161210530:86 , srv1182994292:86 , srv1624272157:86 , srv1697378383:86 , srv1867433268:86 , srv194447655:86 , srv2103672311:86 , srv230652135:86 , srv265914539:86 , srv474865244:86 , srv47918257:86 , srv703565032:86 , srv826825372:86 , srv1120213467:87 , srv1188270837:87 , srv1218384490:87 , srv127913181:87 , srv1310522606:87 , srv14908826:87 , srv1622867008:87 , srv1846569281:87 , srv1858165255:87 , srv1993428690:87 , srv2091282872:87 , srv714531569:87 , srv733149896:87 , srv1019032799:88 , srv1482124747:88 , srv1569115301:88 , srv1792629538:88 , srv258153003:88 , srv821141:88 , srv1504767708:89 , srv252367900:89 , srv398220342:89 , srv673251282:89 , srv854238996:89 , srv1019705160:90 , srv1740077706:90 , srv24468510:90 , srv476325939:90 , srv115168499:91 , srv1570460137:91 , srv943395902:91 , srv892172166:92 , srv861246482:539 } 2023-12-02 02:02:34,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-02 02:02:34,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv369883873=70, srv194447655=51, srv625031552=83, srv1624272157=32, srv854238996=92, srv1597344633=30, srv1188270837=9, srv153574395=26, srv246236393=65, srv1261294239=12, srv442535629=75, srv1019705160=1, srv673251282=85, srv2016747821=55, srv733149896=88, srv1504767708=23, srv1891961830=50, srv1525104389=25, srv1379277637=17, srv47918257=81, srv262247760=68, srv826825372=91, srv1074664479=4, srv1697378383=35, srv1744057218=38, srv2091282872=59, srv398220342=72, srv1784705839=41, srv1810279343=43, srv1987086198=53, srv175381939=40, srv1833571904=44, srv1219213250=11, srv1383195097=18, srv474865244=78, srv1310522606=14, srv861246482=93, srv1993428690=54, srv1182994292=8, srv265914539=69, srv1690760344=33, srv1622867008=31, srv1740077706=36, srv2086075993=58, srv1019032799=0, srv216949931=61, srv1483293953=21, srv464426028=77, srv14908826=22, srv1846569281=46, srv497648006=82, srv1858165255=48, srv1029472809=2, srv233797076=63, srv1120213467=5, srv1161210530=7, srv2075500790=57, srv1539463812=27, srv2072700207=56, srv1508624499=24, srv476325939=79, srv892172166=95, srv127913181=13, srv1740290398=37, srv7725176=89, srv24468510=64, srv999971952=99, srv882600871=94, srv412601433=73, srv252367900=66, srv1748222084=39, srv1792629538=42, srv451696727=76, srv1057555842=3, srv2103672311=60, srv43154857=74, srv258153003=67, srv1482124747=20, srv1867433268=49, srv1958642984=52, srv1218384490=10, srv821141=90, srv940989308=97, srv1843373500=45, srv478096288=80, srv230652135=62, srv379136620=71, srv1569115301=28, srv1320966798=15, srv1354733259=16, srv659962339=84, srv1570460137=29, srv1692305515=34, srv1467043269=19, srv714531569=87, srv921022688=96, srv115168499=6, srv1851902294=47, srv703565032=86, srv943395902=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-02 02:02:34,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-02 02:02:34,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-02 02:02:34,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-02 02:02:34,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-02 02:02:34,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-02 02:02:34,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-02 02:02:34,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-02 02:02:34,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-02 02:02:34,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-02 02:02:34,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-02 02:02:34,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-02 02:02:34,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-02 02:02:34,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-02 02:02:34,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-02 02:02:34,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-02 02:02:34,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-02 02:02:34,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-02 02:02:34,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-02 02:02:34,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-02 02:02:34,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-02 02:02:34,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-02 02:02:34,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-02 02:02:34,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-02 02:02:34,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-02 02:02:34,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-02 02:02:34,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-02 02:02:34,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-02 02:02:34,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-02 02:02:34,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-02 02:02:34,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-02 02:02:34,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-02 02:02:34,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-02 02:02:34,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-02 02:02:34,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-02 02:02:34,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-02 02:02:34,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-02 02:02:34,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-02 02:02:34,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-02 02:02:34,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-02 02:02:34,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-02 02:02:34,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-02 02:02:34,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-02 02:02:34,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-02 02:02:34,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-02 02:02:34,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-02 02:02:34,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-02 02:02:34,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-02 02:02:34,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-02 02:02:34,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-02 02:02:34,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-02 02:02:34,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-02 02:02:34,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-02 02:02:34,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-02 02:02:34,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-02 02:02:34,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-02 02:02:34,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-02 02:02:34,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-02 02:02:34,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-02 02:02:34,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-02 02:02:34,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-02 02:02:34,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-02 02:02:34,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-02 02:02:34,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-02 02:02:34,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-02 02:02:34,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-02 02:02:34,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-02 02:02:34,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-02 02:02:34,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-02 02:02:34,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-02 02:02:34,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-02 02:02:34,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-02 02:02:34,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-02 02:02:34,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-02 02:02:34,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-02 02:02:34,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-02 02:02:34,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-02 02:02:34,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-02 02:02:34,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-02 02:02:34,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-02 02:02:34,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-02 02:02:34,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-02 02:02:34,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-02 02:02:34,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-02 02:02:34,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-02 02:02:34,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-02 02:02:34,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-02 02:02:34,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-02 02:02:34,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-02 02:02:34,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-02 02:02:34,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-02 02:02:34,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-02 02:02:34,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-02 02:02:34,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-02 02:02:34,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-02 02:02:34,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-02 02:02:34,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-02 02:02:34,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-02 02:02:34,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-02 02:02:34,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-02 02:02:34,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-02 02:02:34,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-02 02:02:34,200 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02541291377541991, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03909182260828038, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02541291377541991, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:02:34,201 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-02 02:02:34,201 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014470218738718696, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02541291377541991, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03909182260828038, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02541291377541991, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-02 02:02:34,356 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 229 ms to try 5000 different iterations. Found a solution that moves 972 regions; Going from a computed imbalance of 0.014470218738718696 to a new imbalance of 7.824069364087895E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007792994574261433); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0089643263564985); MoveCostFunction : (multiplier=7.0, imbalance=0.108, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007792994574261433); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:02:34,358 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1570460137:80 , srv127913181:82 , srv1218384490:83 , srv1379277637:83 , srv153574395:83 , srv398220342:83 , srv714531569:83 , srv892172166:83 , srv1029472809:84 , srv115168499:84 , srv1188270837:84 , srv1219213250:84 , srv1261294239:84 , srv14908826:84 , srv1504767708:84 , srv1539463812:84 , srv1784705839:84 , srv1958642984:84 , srv451696727:84 , srv7725176:84 , srv854238996:84 , srv1019032799:85 , srv1019705160:85 , srv1120213467:85 , srv1161210530:85 , srv1182994292:85 , srv1320966798:85 , srv1467043269:85 , srv1482124747:85 , srv1597344633:85 , srv1697378383:85 , srv1740077706:85 , srv1810279343:85 , srv1833571904:85 , srv1843373500:85 , srv1867433268:85 , srv194447655:85 , srv2091282872:85 , srv252367900:85 , srv379136620:85 , srv625031552:85 , srv673251282:85 , srv733149896:85 , srv821141:85 , srv826825372:85 , srv921022688:85 , srv999971952:85 , srv1057555842:86 , srv1074664479:86 , srv1354733259:86 , srv1508624499:86 , srv1569115301:86 , srv1622867008:86 , srv1624272157:86 , srv1740290398:86 , srv175381939:86 , srv1792629538:86 , srv1846569281:86 , srv1858165255:86 , srv1891961830:86 , srv1987086198:86 , srv1993428690:86 , srv2016747821:86 , srv2086075993:86 , srv230652135:86 , srv24468510:86 , srv246236393:86 , srv258153003:86 , srv442535629:86 , srv474865244:86 , srv476325939:86 , srv47918257:86 , srv1310522606:87 , srv1383195097:87 , srv1483293953:87 , srv1690760344:87 , srv1748222084:87 , srv1851902294:87 , srv2072700207:87 , srv216949931:87 , srv233797076:87 , srv265914539:87 , srv412601433:87 , srv464426028:87 , srv478096288:87 , srv497648006:87 , srv703565032:87 , srv940989308:87 , srv943395902:87 , srv1525104389:88 , srv1692305515:88 , srv1744057218:88 , srv262247760:88 , srv369883873:88 , srv43154857:88 , srv659962339:88 , srv882600871:88 , srv2075500790:89 , srv2103672311:89 , srv861246482:527 } 2023-12-02 02:02:34,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-02 02:02:34,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv369883873=70, srv194447655=51, srv625031552=83, srv1624272157=32, srv854238996=92, srv1597344633=30, srv1188270837=9, srv153574395=26, srv246236393=65, srv1261294239=12, srv442535629=75, srv1019705160=1, srv673251282=85, srv2016747821=55, srv733149896=88, srv1504767708=23, srv1891961830=50, srv1525104389=25, srv1379277637=17, srv47918257=81, srv262247760=68, srv826825372=91, srv1074664479=4, srv1697378383=35, srv1744057218=38, srv2091282872=59, srv398220342=72, srv1784705839=41, srv1810279343=43, srv1987086198=53, srv175381939=40, srv1833571904=44, srv1219213250=11, srv1383195097=18, srv474865244=78, srv1310522606=14, srv861246482=93, srv1993428690=54, srv1182994292=8, srv265914539=69, srv1690760344=33, srv1622867008=31, srv1740077706=36, srv2086075993=58, srv1019032799=0, srv216949931=61, srv1483293953=21, srv464426028=77, srv14908826=22, srv1846569281=46, srv497648006=82, srv1858165255=48, srv1029472809=2, srv233797076=63, srv1120213467=5, srv1161210530=7, srv2075500790=57, srv1539463812=27, srv2072700207=56, srv1508624499=24, srv476325939=79, srv892172166=95, srv127913181=13, srv1740290398=37, srv7725176=89, srv24468510=64, srv999971952=99, srv882600871=94, srv412601433=73, srv252367900=66, srv1748222084=39, srv1792629538=42, srv451696727=76, srv1057555842=3, srv2103672311=60, srv43154857=74, srv258153003=67, srv1482124747=20, srv1867433268=49, srv1958642984=52, srv1218384490=10, srv821141=90, srv940989308=97, srv1843373500=45, srv478096288=80, srv230652135=62, srv379136620=71, srv1569115301=28, srv1320966798=15, srv1354733259=16, srv659962339=84, srv1570460137=29, srv1692305515=34, srv1467043269=19, srv714531569=87, srv921022688=96, srv115168499=6, srv1851902294=47, srv703565032=86, srv943395902=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-02 02:02:34,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-02 02:02:34,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-02 02:02:34,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-02 02:02:34,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-02 02:02:34,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-02 02:02:34,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-02 02:02:34,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-02 02:02:34,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-02 02:02:34,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-02 02:02:34,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-02 02:02:34,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-02 02:02:34,457 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007792994574261433); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0089643263564985); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007792994574261433); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:02:34,457 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-02 02:02:34,457 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.756008469393224E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007792994574261433); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0089643263564985); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007792994574261433); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-02 02:02:34,748 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 363 ms to try 5000 different iterations. Found a solution that moves 268 regions; Going from a computed imbalance of 7.756008469393224E-4 to a new imbalance of 7.496369584852491E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.029777777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=6.666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:02:34,748 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv943395902:82 , srv1383195097:83 , srv1833571904:83 , srv369883873:83 , srv659962339:83 , srv1019705160:84 , srv1057555842:84 , srv1182994292:84 , srv127913181:84 , srv1354733259:84 , srv1379277637:84 , srv1483293953:84 , srv14908826:84 , srv1525104389:84 , srv153574395:84 , srv1622867008:84 , srv1690760344:84 , srv1692305515:84 , srv1740290398:84 , srv1744057218:84 , srv1784705839:84 , srv1846569281:84 , srv1851902294:84 , srv1858165255:84 , srv1891961830:84 , srv2016747821:84 , srv2072700207:84 , srv2075500790:84 , srv2103672311:84 , srv216949931:84 , srv24468510:84 , srv252367900:84 , srv262247760:84 , srv476325939:84 , srv47918257:84 , srv497648006:84 , srv625031552:84 , srv826825372:84 , srv882600871:84 , srv1019032799:85 , srv1029472809:85 , srv1120213467:85 , srv115168499:85 , srv1161210530:85 , srv1188270837:85 , srv1218384490:85 , srv1219213250:85 , srv1261294239:85 , srv1310522606:85 , srv1320966798:85 , srv1467043269:85 , srv1504767708:85 , srv1508624499:85 , srv1539463812:85 , srv1570460137:85 , srv1697378383:85 , srv1740077706:85 , srv1748222084:85 , srv1792629538:85 , srv1843373500:85 , srv194447655:85 , srv1987086198:85 , srv1993428690:85 , srv2086075993:85 , srv2091282872:85 , srv230652135:85 , srv233797076:85 , srv246236393:85 , srv265914539:85 , srv379136620:85 , srv398220342:85 , srv43154857:85 , srv442535629:85 , srv451696727:85 , srv464426028:85 , srv474865244:85 , srv478096288:85 , srv673251282:85 , srv703565032:85 , srv714531569:85 , srv733149896:85 , srv821141:85 , srv854238996:85 , srv892172166:85 , srv940989308:85 , srv1074664479:86 , srv1569115301:86 , srv1597344633:86 , srv1624272157:86 , srv175381939:86 , srv1810279343:86 , srv1867433268:86 , srv1958642984:86 , srv258153003:86 , srv412601433:86 , srv7725176:86 , srv921022688:86 , srv999971952:86 , srv1482124747:87 , srv861246482:615 } 2023-12-02 02:02:34,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-02 02:02:34,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv369883873=70, srv194447655=51, srv625031552=83, srv1624272157=32, srv854238996=92, srv1597344633=30, srv1188270837=9, srv153574395=26, srv246236393=65, srv1261294239=12, srv442535629=75, srv1019705160=1, srv673251282=85, srv2016747821=55, srv733149896=88, srv1504767708=23, srv1891961830=50, srv1525104389=25, srv1379277637=17, srv47918257=81, srv262247760=68, srv826825372=91, srv1074664479=4, srv1697378383=35, srv1744057218=38, srv2091282872=59, srv398220342=72, srv1784705839=41, srv1810279343=43, srv1987086198=53, srv175381939=40, srv1833571904=44, srv1219213250=11, srv1383195097=18, srv474865244=78, srv1310522606=14, srv861246482=93, srv1993428690=54, srv1182994292=8, srv265914539=69, srv1690760344=33, srv1622867008=31, srv1740077706=36, srv2086075993=58, srv1019032799=0, srv216949931=61, srv1483293953=21, srv464426028=77, srv14908826=22, srv1846569281=46, srv497648006=82, srv1858165255=48, srv1029472809=2, srv233797076=63, srv1120213467=5, srv1161210530=7, srv2075500790=57, srv1539463812=27, srv2072700207=56, srv1508624499=24, srv476325939=79, srv892172166=95, srv127913181=13, srv1740290398=37, srv7725176=89, srv24468510=64, srv999971952=99, srv882600871=94, srv412601433=73, srv252367900=66, srv1748222084=39, srv1792629538=42, srv451696727=76, srv1057555842=3, srv2103672311=60, srv43154857=74, srv258153003=67, srv1482124747=20, srv1867433268=49, srv1958642984=52, srv1218384490=10, srv821141=90, srv940989308=97, srv1843373500=45, srv478096288=80, srv230652135=62, srv379136620=71, srv1569115301=28, srv1320966798=15, srv1354733259=16, srv659962339=84, srv1570460137=29, srv1692305515=34, srv1467043269=19, srv714531569=87, srv921022688=96, srv115168499=6, srv1851902294=47, srv703565032=86, srv943395902=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-02 02:02:34,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-02 02:02:34,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-02 02:02:34,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-02 02:02:34,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-02 02:02:34,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-02 02:02:34,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-02 02:02:34,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-02 02:02:34,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-02 02:02:34,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-02 02:02:34,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-02 02:02:34,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-02 02:02:34,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-02 02:02:34,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-02 02:02:34,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-02 02:02:34,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-02 02:02:34,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-02 02:02:34,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-02 02:02:34,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-02 02:02:34,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-02 02:02:34,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-02 02:02:34,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-02 02:02:34,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-02 02:02:34,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-02 02:02:34,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-02 02:02:34,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-02 02:02:34,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-02 02:02:34,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-02 02:02:34,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-02 02:02:34,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-02 02:02:34,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-02 02:02:34,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-02 02:02:34,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-02 02:02:34,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-02 02:02:34,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-02 02:02:34,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-02 02:02:34,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-02 02:02:34,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-02 02:02:34,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-02 02:02:34,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-02 02:02:34,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-02 02:02:34,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-02 02:02:34,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-02 02:02:34,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-02 02:02:34,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-02 02:02:34,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-02 02:02:34,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-02 02:02:34,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-02 02:02:34,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-02 02:02:34,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-02 02:02:34,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-02 02:02:34,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-02 02:02:34,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-02 02:02:34,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-02 02:02:34,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-02 02:02:34,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-02 02:02:34,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-02 02:02:34,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-02 02:02:34,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-02 02:02:34,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-02 02:02:34,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-02 02:02:34,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-02 02:02:34,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-02 02:02:34,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-02 02:02:34,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-02 02:02:34,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-02 02:02:34,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-02 02:02:34,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-02 02:02:34,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-02 02:02:34,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-02 02:02:34,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-02 02:02:34,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-02 02:02:34,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-02 02:02:34,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-02 02:02:34,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-02 02:02:34,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-02 02:02:34,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-02 02:02:34,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-02 02:02:34,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-02 02:02:34,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-02 02:02:34,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-02 02:02:34,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-02 02:02:34,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-02 02:02:34,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-02 02:02:34,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-02 02:02:34,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-02 02:02:34,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-02 02:02:34,847 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=6.666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:02:34,847 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-02 02:02:34,847 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.30871197396595E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=6.666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-02 02:02:34,998 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 221 ms to try 5000 different iterations. Found a solution that moves 48 regions; Going from a computed imbalance of 7.30871197396595E-5 to a new imbalance of 7.409342480006837E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.005333333333333333); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:02:34,998 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1310522606:83 , srv1482124747:83 , srv1569115301:83 , srv1624272157:83 , srv1740077706:83 , srv2091282872:83 , srv476325939:83 , srv478096288:83 , srv703565032:83 , srv1019032799:84 , srv1029472809:84 , srv1057555842:84 , srv1074664479:84 , srv1120213467:84 , srv115168499:84 , srv1161210530:84 , srv1182994292:84 , srv1188270837:84 , srv1218384490:84 , srv1219213250:84 , srv1261294239:84 , srv1320966798:84 , srv1354733259:84 , srv1383195097:84 , srv1467043269:84 , srv14908826:84 , srv1504767708:84 , srv1508624499:84 , srv1525104389:84 , srv153574395:84 , srv1539463812:84 , srv1570460137:84 , srv1597344633:84 , srv1622867008:84 , srv1690760344:84 , srv1692305515:84 , srv1697378383:84 , srv1740290398:84 , srv1744057218:84 , srv1748222084:84 , srv175381939:84 , srv1792629538:84 , srv1810279343:84 , srv1846569281:84 , srv1851902294:84 , srv1858165255:84 , srv1867433268:84 , srv1891961830:84 , srv194447655:84 , srv1958642984:84 , srv1987086198:84 , srv1993428690:84 , srv2016747821:84 , srv2072700207:84 , srv2075500790:84 , srv2086075993:84 , srv2103672311:84 , srv216949931:84 , srv230652135:84 , srv233797076:84 , srv24468510:84 , srv246236393:84 , srv252367900:84 , srv258153003:84 , srv262247760:84 , srv265914539:84 , srv369883873:84 , srv379136620:84 , srv398220342:84 , srv412601433:84 , srv43154857:84 , srv442535629:84 , srv451696727:84 , srv464426028:84 , srv474865244:84 , srv47918257:84 , srv497648006:84 , srv659962339:84 , srv673251282:84 , srv714531569:84 , srv733149896:84 , srv7725176:84 , srv821141:84 , srv826825372:84 , srv854238996:84 , srv882600871:84 , srv892172166:84 , srv921022688:84 , srv940989308:84 , srv943395902:84 , srv999971952:84 , srv1019705160:85 , srv127913181:85 , srv1379277637:85 , srv1483293953:85 , srv1784705839:85 , srv1843373500:85 , srv625031552:85 , srv1833571904:86 , srv861246482:684 } 2023-12-02 02:02:35,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-02 02:02:35,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv369883873=70, srv194447655=51, srv625031552=83, srv1624272157=32, srv854238996=92, srv1597344633=30, srv1188270837=9, srv153574395=26, srv246236393=65, srv1261294239=12, srv442535629=75, srv1019705160=1, srv673251282=85, srv2016747821=55, srv733149896=88, srv1504767708=23, srv1891961830=50, srv1525104389=25, srv1379277637=17, srv47918257=81, srv262247760=68, srv826825372=91, srv1074664479=4, srv1697378383=35, srv1744057218=38, srv2091282872=59, srv398220342=72, srv1784705839=41, srv1810279343=43, srv1987086198=53, srv175381939=40, srv1833571904=44, srv1219213250=11, srv1383195097=18, srv474865244=78, srv1310522606=14, srv861246482=93, srv1993428690=54, srv1182994292=8, srv265914539=69, srv1690760344=33, srv1622867008=31, srv1740077706=36, srv2086075993=58, srv1019032799=0, srv216949931=61, srv1483293953=21, srv464426028=77, srv14908826=22, srv1846569281=46, srv497648006=82, srv1858165255=48, srv1029472809=2, srv233797076=63, srv1120213467=5, srv1161210530=7, srv2075500790=57, srv1539463812=27, srv2072700207=56, srv1508624499=24, srv476325939=79, srv892172166=95, srv127913181=13, srv1740290398=37, srv7725176=89, srv24468510=64, srv999971952=99, srv882600871=94, srv412601433=73, srv252367900=66, srv1748222084=39, srv1792629538=42, srv451696727=76, srv1057555842=3, srv2103672311=60, srv43154857=74, srv258153003=67, srv1482124747=20, srv1867433268=49, srv1958642984=52, srv1218384490=10, srv821141=90, srv940989308=97, srv1843373500=45, srv478096288=80, srv230652135=62, srv379136620=71, srv1569115301=28, srv1320966798=15, srv1354733259=16, srv659962339=84, srv1570460137=29, srv1692305515=34, srv1467043269=19, srv714531569=87, srv921022688=96, srv115168499=6, srv1851902294=47, srv703565032=86, srv943395902=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-02 02:02:35,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-02 02:02:35,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-02 02:02:35,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-02 02:02:35,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-02 02:02:35,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-02 02:02:35,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-02 02:02:35,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-02 02:02:35,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-02 02:02:35,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-02 02:02:35,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-02 02:02:35,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-02 02:02:35,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-02 02:02:35,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-02 02:02:35,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-02 02:02:35,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-02 02:02:35,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-02 02:02:35,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-02 02:02:35,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-02 02:02:35,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-02 02:02:35,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-02 02:02:35,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-02 02:02:35,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-02 02:02:35,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-02 02:02:35,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-02 02:02:35,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-02 02:02:35,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-02 02:02:35,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-02 02:02:35,095 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:02:35,095 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1310522606:83 , srv1482124747:83 , srv1569115301:83 , srv1624272157:83 , srv1740077706:83 , srv2091282872:83 , srv476325939:83 , srv478096288:83 , srv703565032:83 , srv1019032799:84 , srv1029472809:84 , srv1057555842:84 , srv1074664479:84 , srv1120213467:84 , srv115168499:84 , srv1161210530:84 , srv1182994292:84 , srv1188270837:84 , srv1218384490:84 , srv1219213250:84 , srv1261294239:84 , srv1320966798:84 , srv1354733259:84 , srv1383195097:84 , srv1467043269:84 , srv14908826:84 , srv1504767708:84 , srv1508624499:84 , srv1525104389:84 , srv153574395:84 , srv1539463812:84 , srv1570460137:84 , srv1597344633:84 , srv1622867008:84 , srv1690760344:84 , srv1692305515:84 , srv1697378383:84 , srv1740290398:84 , srv1744057218:84 , srv1748222084:84 , srv175381939:84 , srv1792629538:84 , srv1810279343:84 , srv1846569281:84 , srv1851902294:84 , srv1858165255:84 , srv1867433268:84 , srv1891961830:84 , srv194447655:84 , srv1958642984:84 , srv1987086198:84 , srv1993428690:84 , srv2016747821:84 , srv2072700207:84 , srv2075500790:84 , srv2086075993:84 , srv2103672311:84 , srv216949931:84 , srv230652135:84 , srv233797076:84 , srv24468510:84 , srv246236393:84 , srv252367900:84 , srv258153003:84 , srv262247760:84 , srv265914539:84 , srv369883873:84 , srv379136620:84 , srv398220342:84 , srv412601433:84 , srv43154857:84 , srv442535629:84 , srv451696727:84 , srv464426028:84 , srv474865244:84 , srv47918257:84 , srv497648006:84 , srv659962339:84 , srv673251282:84 , srv714531569:84 , srv733149896:84 , srv7725176:84 , srv821141:84 , srv826825372:84 , srv854238996:84 , srv882600871:84 , srv892172166:84 , srv921022688:84 , srv940989308:84 , srv943395902:84 , srv999971952:84 , srv1019705160:85 , srv127913181:85 , srv1379277637:85 , srv1483293953:85 , srv1784705839:85 , srv1843373500:85 , srv625031552:85 , srv1833571904:86 , srv861246482:684 } 2023-12-02 02:02:35,118 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=542 (was 501) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 166) - ProcessCount LEAK? -, AvailableMemoryMB=5979 (was 6461) 2023-12-02 02:02:35,126 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=542, ProcessCount=168, AvailableMemoryMB=5978 2023-12-02 02:02:35,126 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-02 02:02:35,126 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-02 02:02:35,137 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1179098512:84 , srv1470909079:84 , srv1501983585:84 , srv1579137735:84 , srv159688813:84 , srv1644057947:84 , srv1692361715:84 , srv1927649710:84 , srv1991273414:84 , srv2041084108:84 , srv2059362412:84 , srv2087598345:84 , srv2095694606:84 , srv2146657514:84 , srv240358475:84 , srv262084792:84 , srv376216584:84 , srv403448403:84 , srv422083356:84 , srv444837277:84 , srv551136982:84 , srv626291215:84 , srv656973803:84 , srv695349323:84 , srv697191743:84 , srv720909671:84 , srv860030039:84 , srv906809437:84 , srv969387961:84 , srv321278644:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-02 02:02:35,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-02 02:02:35,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv240358475=14, srv376216584=17, srv551136982=21, srv656973803=23, srv1644057947=5, srv422083356=19, srv1501983585=2, srv2041084108=9, srv906809437=28, srv444837277=20, srv695349323=24, srv1991273414=8, srv262084792=15, srv969387961=29, srv697191743=25, srv2059362412=10, srv2087598345=11, srv2095694606=12, srv1927649710=7, srv1579137735=3, srv1470909079=1, srv626291215=22, srv860030039=27, srv159688813=4, srv720909671=26, srv403448403=18, srv1179098512=0, srv1692361715=6, srv2146657514=13, srv321278644=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-02 02:02:35,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-02 02:02:35,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-02 02:02:35,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-02 02:02:35,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-02 02:02:35,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-02 02:02:35,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-02 02:02:35,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-02 02:02:35,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-02 02:02:35,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-02 02:02:35,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-02 02:02:35,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-02 02:02:35,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-02 02:02:35,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-02 02:02:35,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-02 02:02:35,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-02 02:02:35,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-02 02:02:35,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-02 02:02:35,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-02 02:02:35,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-02 02:02:35,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-02 02:02:35,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-02 02:02:35,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-02 02:02:35,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-02 02:02:35,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-02 02:02:35,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-02 02:02:35,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-02 02:02:35,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-02 02:02:35,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-02 02:02:35,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-02 02:02:35,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-02 02:02:35,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-02 02:02:35,175 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1797202682201597, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:02:35,175 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912759335961034, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1797202682201597, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-02 02:04:35,157 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9111838 different iterations. Found a solution that moves 1852 regions; Going from a computed imbalance of 0.9912759335961034 to a new imbalance of 4.322660390073086E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6859259259259259, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:04:35,163 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1179098512:90 , srv1470909079:90 , srv1501983585:90 , srv1579137735:90 , srv159688813:90 , srv1644057947:90 , srv1692361715:90 , srv1927649710:90 , srv1991273414:90 , srv2041084108:90 , srv2059362412:90 , srv2087598345:90 , srv2095694606:90 , srv2146657514:90 , srv240358475:90 , srv262084792:90 , srv321278644:90 , srv376216584:90 , srv403448403:90 , srv422083356:90 , srv444837277:90 , srv551136982:90 , srv626291215:90 , srv656973803:90 , srv695349323:90 , srv697191743:90 , srv720909671:90 , srv860030039:90 , srv906809437:90 , srv969387961:90 } 2023-12-02 02:04:35,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-02 02:04:35,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv240358475=14, srv376216584=17, srv551136982=21, srv656973803=23, srv1644057947=5, srv422083356=19, srv1501983585=2, srv2041084108=9, srv906809437=28, srv444837277=20, srv695349323=24, srv1991273414=8, srv262084792=15, srv969387961=29, srv697191743=25, srv2059362412=10, srv2087598345=11, srv2095694606=12, srv1927649710=7, srv1579137735=3, srv1470909079=1, srv626291215=22, srv860030039=27, srv159688813=4, srv720909671=26, srv403448403=18, srv1179098512=0, srv1692361715=6, srv2146657514=13, srv321278644=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-02 02:04:35,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-02 02:04:35,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-02 02:04:35,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-02 02:04:35,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-02 02:04:35,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-02 02:04:35,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-02 02:04:35,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-02 02:04:35,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-02 02:04:35,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-02 02:04:35,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-02 02:04:35,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-02 02:04:35,180 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-02 02:04:35,180 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1179098512:90 , srv1470909079:90 , srv1501983585:90 , srv1579137735:90 , srv159688813:90 , srv1644057947:90 , srv1692361715:90 , srv1927649710:90 , srv1991273414:90 , srv2041084108:90 , srv2059362412:90 , srv2087598345:90 , srv2095694606:90 , srv2146657514:90 , srv240358475:90 , srv262084792:90 , srv321278644:90 , srv376216584:90 , srv403448403:90 , srv422083356:90 , srv444837277:90 , srv551136982:90 , srv626291215:90 , srv656973803:90 , srv695349323:90 , srv697191743:90 , srv720909671:90 , srv860030039:90 , srv906809437:90 , srv969387961:90 } 2023-12-02 02:04:35,191 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=276 (was 542), ProcessCount=167 (was 168), AvailableMemoryMB=5749 (was 5978)