2023-03-31 16:55:54,288 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-03-31 16:55:54,679 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 16:55:54,819 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-31 16:55:54,898 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-31 16:55:54,907 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-31 16:55:54,924 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=454, ProcessCount=169, AvailableMemoryMB=4508 2023-03-31 16:55:54,927 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-31 16:55:54,928 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-31 16:55:55,097 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1029541511:84 , srv1041161441:84 , srv1071458509:84 , srv1117237659:84 , srv1142933720:84 , srv114398957:84 , srv117777099:84 , srv1184606482:84 , srv1184941823:84 , srv1203818828:84 , srv1208890389:84 , srv1221386077:84 , srv1234176113:84 , srv1234564289:84 , srv1287846643:84 , srv1297953227:84 , srv1330916163:84 , srv1408368030:84 , srv1434779176:84 , srv147207403:84 , srv1523066739:84 , srv1569330356:84 , srv1581020452:84 , srv1597799559:84 , srv1604877780:84 , srv1614521090:84 , srv1637063933:84 , srv1648022299:84 , srv1657527376:84 , srv1662268580:84 , srv1686407390:84 , srv168673422:84 , srv1688603373:84 , srv1698511201:84 , srv1723589500:84 , srv1742176848:84 , srv1747394150:84 , srv1797570544:84 , srv1828779618:84 , srv1843653218:84 , srv1848161103:84 , srv1851885987:84 , srv1853603965:84 , srv1860722003:84 , srv1871541119:84 , srv1881958787:84 , srv1913161006:84 , srv192132763:84 , srv1930250262:84 , srv1939209424:84 , srv1944846234:84 , srv1951821118:84 , srv1965186536:84 , srv1995196997:84 , srv2008119661:84 , srv2031148794:84 , srv2032571174:84 , srv2063463830:84 , srv2096418030:84 , srv2123668190:84 , srv2132919177:84 , srv23039778:84 , srv230874629:84 , srv266073327:84 , srv292744661:84 , srv345694099:84 , srv34927053:84 , srv35367966:84 , srv358085149:84 , srv358617637:84 , srv360597693:84 , srv373779082:84 , srv398700446:84 , srv415664514:84 , srv472153332:84 , srv524208954:84 , srv525443426:84 , srv537097432:84 , srv552168170:84 , srv560362471:84 , srv579920910:84 , srv637340971:84 , srv641676810:84 , srv672030925:84 , srv675025885:84 , srv720080085:84 , srv728918563:84 , srv763034966:84 , srv796940827:84 , srv800688241:84 , srv813084598:84 , srv826419682:84 , srv828668929:84 , srv888336199:84 , srv907348287:84 , srv951213773:84 , srv95328671:84 , srv990166502:84 , srv994902771:84 , srv1018984439:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-03-31 16:55:55,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 16:55:55,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv415664514=74, srv524208954=76, srv1208890389=11, srv1071458509=3, srv1234176113=13, srv230874629=63, srv1657527376=29, srv1853603965=43, srv2132919177=61, srv1637063933=27, srv1871541119=45, srv1434779176=19, srv1698511201=34, srv537097432=78, srv990166502=98, srv1029541511=1, srv907348287=95, srv1688603373=33, srv358617637=70, srv951213773=96, srv1184606482=8, srv1604877780=25, srv637340971=82, srv2031148794=56, srv35367966=68, srv1965186536=53, srv1747394150=37, srv168673422=32, srv292744661=65, srv360597693=71, srv1851885987=42, srv1944846234=51, srv1184941823=9, srv345694099=66, srv1041161441=2, srv1221386077=12, srv1297953227=16, srv1913161006=47, srv672030925=84, srv1287846643=15, srv192132763=48, srv2123668190=60, srv1686407390=31, srv1569330356=22, srv579920910=81, srv1860722003=44, srv373779082=72, srv1142933720=5, srv117777099=7, srv796940827=89, srv114398957=6, srv641676810=83, srv1723589500=35, srv1843653218=40, srv720080085=86, srv1995196997=54, srv1581020452=23, srv888336199=94, srv1203818828=10, srv1848161103=41, srv358085149=69, srv1408368030=18, srv1523066739=21, srv828668929=93, srv1662268580=30, srv1797570544=38, srv813084598=91, srv826419682=92, srv1828779618=39, srv2008119661=55, srv560362471=80, srv2032571174=57, srv552168170=79, srv472153332=75, srv2063463830=58, srv1742176848=36, srv994902771=99, srv34927053=67, srv1614521090=26, srv1117237659=4, srv1597799559=24, srv23039778=62, srv95328671=97, srv728918563=87, srv675025885=85, srv147207403=20, srv1234564289=14, srv398700446=73, srv1018984439=0, srv1330916163=17, srv2096418030=59, srv266073327=64, srv1881958787=46, srv1930250262=49, srv800688241=90, srv1648022299=28, srv525443426=77, srv763034966=88, srv1939209424=50, srv1951821118=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 16:55:55,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 16:55:55,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 16:55:55,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 16:55:55,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 16:55:55,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 16:55:55,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 16:55:55,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 16:55:55,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 16:55:55,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 16:55:55,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 16:55:55,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 16:55:55,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 16:55:55,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 16:55:55,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 16:55:55,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 16:55:55,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 16:55:55,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 16:55:55,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 16:55:55,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 16:55:55,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 16:55:55,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 16:55:55,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 16:55:55,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 16:55:55,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 16:55:55,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 16:55:55,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 16:55:55,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 16:55:55,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 16:55:55,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 16:55:55,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 16:55:55,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-31 16:55:55,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-31 16:55:55,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-31 16:55:55,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-31 16:55:55,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-31 16:55:55,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-31 16:55:55,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-31 16:55:55,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-31 16:55:55,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-31 16:55:55,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-31 16:55:55,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-31 16:55:55,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-31 16:55:55,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-31 16:55:55,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-31 16:55:55,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-31 16:55:55,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-31 16:55:55,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-31 16:55:55,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-31 16:55:55,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-31 16:55:55,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-31 16:55:55,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-31 16:55:55,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-31 16:55:55,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-31 16:55:55,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-31 16:55:55,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-31 16:55:55,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-31 16:55:55,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-31 16:55:55,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-31 16:55:55,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-31 16:55:55,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-31 16:55:55,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-31 16:55:55,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-31 16:55:55,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-31 16:55:55,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-31 16:55:55,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-31 16:55:55,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-31 16:55:55,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-31 16:55:55,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-31 16:55:55,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-31 16:55:55,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-31 16:55:55,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-31 16:55:55,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-31 16:55:55,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-31 16:55:55,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-31 16:55:55,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-31 16:55:55,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-31 16:55:55,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-31 16:55:55,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-31 16:55:55,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-31 16:55:55,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-31 16:55:55,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-31 16:55:55,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-31 16:55:55,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-31 16:55:55,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-31 16:55:55,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-31 16:55:55,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-31 16:55:55,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-31 16:55:55,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-31 16:55:55,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-31 16:55:55,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-31 16:55:55,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-31 16:55:55,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-31 16:55:55,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-31 16:55:55,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-31 16:55:55,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-31 16:55:55,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-31 16:55:55,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-31 16:55:55,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-31 16:55:55,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-31 16:55:55,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-31 16:55:55,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-31 16:55:55,742 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:55:55,742 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-31 16:55:55,743 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-31 16:55:56,679 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1403 ms to try 5000 different iterations. Found a solution that moves 4659 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18354497585063617. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038149619140675904, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05884980597503838, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5176666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038149619140675904, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17866666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:55:56,715 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv524208954:81 , srv1657527376:83 , srv1828779618:83 , srv23039778:83 , srv345694099:83 , srv826419682:83 , srv1041161441:84 , srv114398957:84 , srv1184941823:84 , srv1203818828:84 , srv1287846643:84 , srv1434779176:84 , srv1597799559:84 , srv1604877780:84 , srv266073327:84 , srv398700446:84 , srv537097432:84 , srv560362471:84 , srv1221386077:85 , srv1330916163:85 , srv147207403:85 , srv1581020452:85 , srv1637063933:85 , srv1698511201:85 , srv1742176848:85 , srv1871541119:85 , srv2008119661:85 , srv2032571174:85 , srv2063463830:85 , srv2096418030:85 , srv2132919177:85 , srv472153332:85 , srv637340971:85 , srv728918563:85 , srv800688241:85 , srv828668929:85 , srv1071458509:86 , srv1117237659:86 , srv1523066739:86 , srv1569330356:86 , srv1614521090:86 , srv1648022299:86 , srv1848161103:86 , srv1860722003:86 , srv1913161006:86 , srv1944846234:86 , srv1965186536:86 , srv1995196997:86 , srv2031148794:86 , srv230874629:86 , srv292744661:86 , srv35367966:86 , srv373779082:86 , srv415664514:86 , srv579920910:86 , srv796940827:86 , srv888336199:86 , srv117777099:87 , srv168673422:87 , srv1843653218:87 , srv192132763:87 , srv1930250262:87 , srv2123668190:87 , srv34927053:87 , srv675025885:87 , srv720080085:87 , srv763034966:87 , srv813084598:87 , srv95328671:87 , srv994902771:87 , srv1029541511:88 , srv1184606482:88 , srv1208890389:88 , srv1234564289:88 , srv1408368030:88 , srv1723589500:88 , srv1797570544:88 , srv1853603965:88 , srv358617637:88 , srv990166502:88 , srv1142933720:89 , srv1234176113:89 , srv1662268580:89 , srv1686407390:89 , srv1688603373:89 , srv360597693:89 , srv552168170:89 , srv641676810:89 , srv907348287:89 , srv1939209424:90 , srv1951821118:90 , srv525443426:90 , srv1747394150:91 , srv1881958787:91 , srv358085149:91 , srv951213773:92 , srv672030925:93 , srv1851885987:95 , srv1297953227:96 , srv1018984439:429 } 2023-03-31 16:55:56,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 16:55:56,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv415664514=74, srv524208954=76, srv1208890389=11, srv1071458509=3, srv1234176113=13, srv230874629=63, srv1657527376=29, srv1853603965=43, srv2132919177=61, srv1637063933=27, srv1871541119=45, srv1434779176=19, srv1698511201=34, srv537097432=78, srv990166502=98, srv1029541511=1, srv907348287=95, srv1688603373=33, srv358617637=70, srv951213773=96, srv1184606482=8, srv1604877780=25, srv637340971=82, srv2031148794=56, srv35367966=68, srv1965186536=53, srv1747394150=37, srv168673422=32, srv292744661=65, srv360597693=71, srv1851885987=42, srv1944846234=51, srv1184941823=9, srv345694099=66, srv1041161441=2, srv1221386077=12, srv1297953227=16, srv1913161006=47, srv672030925=84, srv1287846643=15, srv192132763=48, srv2123668190=60, srv1686407390=31, srv1569330356=22, srv579920910=81, srv1860722003=44, srv373779082=72, srv1142933720=5, srv117777099=7, srv796940827=89, srv114398957=6, srv641676810=83, srv1723589500=35, srv1843653218=40, srv720080085=86, srv1995196997=54, srv1581020452=23, srv888336199=94, srv1203818828=10, srv1848161103=41, srv358085149=69, srv1408368030=18, srv1523066739=21, srv828668929=93, srv1662268580=30, srv1797570544=38, srv813084598=91, srv826419682=92, srv1828779618=39, srv2008119661=55, srv560362471=80, srv2032571174=57, srv552168170=79, srv472153332=75, srv2063463830=58, srv1742176848=36, srv994902771=99, srv34927053=67, srv1614521090=26, srv1117237659=4, srv1597799559=24, srv23039778=62, srv95328671=97, srv728918563=87, srv675025885=85, srv147207403=20, srv1234564289=14, srv398700446=73, srv1018984439=0, srv1330916163=17, srv2096418030=59, srv266073327=64, srv1881958787=46, srv1930250262=49, srv800688241=90, srv1648022299=28, srv525443426=77, srv763034966=88, srv1939209424=50, srv1951821118=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 16:55:56,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 16:55:56,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 16:55:56,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 16:55:56,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 16:55:56,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 16:55:56,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 16:55:56,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 16:55:56,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 16:55:56,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 16:55:56,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 16:55:56,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 16:55:56,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 16:55:56,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 16:55:56,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 16:55:56,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 16:55:56,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 16:55:56,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 16:55:56,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 16:55:56,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 16:55:56,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 16:55:56,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 16:55:56,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 16:55:56,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 16:55:56,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 16:55:56,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 16:55:56,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 16:55:56,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 16:55:56,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 16:55:56,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 16:55:56,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 16:55:56,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-31 16:55:56,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-31 16:55:56,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-31 16:55:56,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-31 16:55:56,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-31 16:55:56,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-31 16:55:56,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-31 16:55:56,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-31 16:55:56,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-31 16:55:56,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-31 16:55:56,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-31 16:55:56,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-31 16:55:56,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-31 16:55:56,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-31 16:55:56,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-31 16:55:56,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-31 16:55:56,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-31 16:55:56,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-31 16:55:56,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-31 16:55:56,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-31 16:55:56,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-31 16:55:56,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-31 16:55:56,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-31 16:55:56,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-31 16:55:56,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-31 16:55:56,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-31 16:55:56,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-31 16:55:56,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-31 16:55:56,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-31 16:55:56,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-31 16:55:56,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-31 16:55:56,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-31 16:55:56,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-31 16:55:56,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-31 16:55:56,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-31 16:55:56,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-31 16:55:56,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-31 16:55:56,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-31 16:55:56,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-31 16:55:56,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-31 16:55:56,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-31 16:55:56,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-31 16:55:56,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-31 16:55:56,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-31 16:55:56,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-31 16:55:56,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-31 16:55:56,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-31 16:55:56,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-31 16:55:56,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-31 16:55:56,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-31 16:55:56,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-31 16:55:56,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-31 16:55:56,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-31 16:55:56,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-31 16:55:56,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-31 16:55:56,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-31 16:55:56,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-31 16:55:56,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-31 16:55:56,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-31 16:55:56,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-31 16:55:56,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-31 16:55:56,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-31 16:55:56,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-31 16:55:56,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-31 16:55:56,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-31 16:55:56,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-31 16:55:56,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-31 16:55:56,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-31 16:55:56,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-31 16:55:56,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-31 16:55:56,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-31 16:55:56,970 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038149619140675904, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05884980597503838, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038149619140675904, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17866666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:55:56,970 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-31 16:55:56,971 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1835123528353705, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038149619140675904, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05884980597503838, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038149619140675904, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17866666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-31 16:55:57,385 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 570 ms to try 5000 different iterations. Found a solution that moves 2694 regions; Going from a computed imbalance of 0.1835123528353705 to a new imbalance of 0.014258610282157805. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019920655680937024); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03376820704388187, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29933333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019920655680937024); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:55:57,399 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv672030925:78 , srv1297953227:79 , srv1851885987:79 , srv1747394150:81 , srv1881958787:81 , srv1951821118:81 , srv358085149:81 , srv525443426:81 , srv1723589500:82 , srv1797570544:82 , srv192132763:82 , srv1939209424:82 , srv358617637:82 , srv951213773:82 , srv1029541511:83 , srv1184606482:83 , srv1234564289:83 , srv360597693:83 , srv552168170:83 , srv641676810:83 , srv675025885:83 , srv907348287:83 , srv1142933720:84 , srv1686407390:84 , srv1688603373:84 , srv1853603965:84 , srv1930250262:84 , srv888336199:84 , srv990166502:84 , srv994902771:84 , srv117777099:85 , srv1234176113:85 , srv1648022299:85 , srv1860722003:85 , srv1944846234:85 , srv1995196997:85 , srv2123668190:85 , srv34927053:85 , srv763034966:85 , srv1203818828:86 , srv1208890389:86 , srv1408368030:86 , srv1523066739:86 , srv1569330356:86 , srv1614521090:86 , srv1965186536:86 , srv2031148794:86 , srv2032571174:86 , srv2132919177:86 , srv230874629:86 , srv637340971:86 , srv720080085:86 , srv728918563:86 , srv813084598:86 , srv826419682:86 , srv95328671:86 , srv1041161441:87 , srv1071458509:87 , srv1117237659:87 , srv1330916163:87 , srv1581020452:87 , srv1604877780:87 , srv1637063933:87 , srv1662268580:87 , srv1698511201:87 , srv1843653218:87 , srv1848161103:87 , srv1913161006:87 , srv2063463830:87 , srv2096418030:87 , srv292744661:87 , srv35367966:87 , srv415664514:87 , srv537097432:87 , srv579920910:87 , srv800688241:87 , srv828668929:87 , srv114398957:88 , srv1184941823:88 , srv1221386077:88 , srv147207403:88 , srv1871541119:88 , srv2008119661:88 , srv345694099:88 , srv398700446:88 , srv472153332:88 , srv796940827:88 , srv1287846643:89 , srv1434779176:89 , srv1597799559:89 , srv1657527376:89 , srv168673422:89 , srv1742176848:89 , srv1828779618:89 , srv266073327:89 , srv373779082:89 , srv23039778:90 , srv560362471:91 , srv524208954:93 , srv1018984439:522 } 2023-03-31 16:55:57,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 16:55:57,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv415664514=74, srv524208954=76, srv1208890389=11, srv1071458509=3, srv1234176113=13, srv230874629=63, srv1657527376=29, srv1853603965=43, srv2132919177=61, srv1637063933=27, srv1871541119=45, srv1434779176=19, srv1698511201=34, srv537097432=78, srv990166502=98, srv1029541511=1, srv907348287=95, srv1688603373=33, srv358617637=70, srv951213773=96, srv1184606482=8, srv1604877780=25, srv637340971=82, srv2031148794=56, srv35367966=68, srv1965186536=53, srv1747394150=37, srv168673422=32, srv292744661=65, srv360597693=71, srv1851885987=42, srv1944846234=51, srv1184941823=9, srv345694099=66, srv1041161441=2, srv1221386077=12, srv1297953227=16, srv1913161006=47, srv672030925=84, srv1287846643=15, srv192132763=48, srv2123668190=60, srv1686407390=31, srv1569330356=22, srv579920910=81, srv1860722003=44, srv373779082=72, srv1142933720=5, srv117777099=7, srv796940827=89, srv114398957=6, srv641676810=83, srv1723589500=35, srv1843653218=40, srv720080085=86, srv1995196997=54, srv1581020452=23, srv888336199=94, srv1203818828=10, srv1848161103=41, srv358085149=69, srv1408368030=18, srv1523066739=21, srv828668929=93, srv1662268580=30, srv1797570544=38, srv813084598=91, srv826419682=92, srv1828779618=39, srv2008119661=55, srv560362471=80, srv2032571174=57, srv552168170=79, srv472153332=75, srv2063463830=58, srv1742176848=36, srv994902771=99, srv34927053=67, srv1614521090=26, srv1117237659=4, srv1597799559=24, srv23039778=62, srv95328671=97, srv728918563=87, srv675025885=85, srv147207403=20, srv1234564289=14, srv398700446=73, srv1018984439=0, srv1330916163=17, srv2096418030=59, srv266073327=64, srv1881958787=46, srv1930250262=49, srv800688241=90, srv1648022299=28, srv525443426=77, srv763034966=88, srv1939209424=50, srv1951821118=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 16:55:57,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 16:55:57,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 16:55:57,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 16:55:57,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 16:55:57,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 16:55:57,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 16:55:57,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 16:55:57,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 16:55:57,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 16:55:57,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 16:55:57,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 16:55:57,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 16:55:57,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 16:55:57,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 16:55:57,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 16:55:57,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 16:55:57,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 16:55:57,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 16:55:57,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 16:55:57,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 16:55:57,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 16:55:57,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 16:55:57,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 16:55:57,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 16:55:57,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 16:55:57,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 16:55:57,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 16:55:57,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 16:55:57,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 16:55:57,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 16:55:57,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-31 16:55:57,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-31 16:55:57,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-31 16:55:57,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-31 16:55:57,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-31 16:55:57,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-31 16:55:57,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-31 16:55:57,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-31 16:55:57,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-31 16:55:57,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-31 16:55:57,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-31 16:55:57,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-31 16:55:57,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-31 16:55:57,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-31 16:55:57,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-31 16:55:57,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-31 16:55:57,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-31 16:55:57,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-31 16:55:57,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-31 16:55:57,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-31 16:55:57,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-31 16:55:57,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-31 16:55:57,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-31 16:55:57,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-31 16:55:57,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-31 16:55:57,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-31 16:55:57,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-31 16:55:57,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-31 16:55:57,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-31 16:55:57,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-31 16:55:57,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-31 16:55:57,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-31 16:55:57,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-31 16:55:57,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-31 16:55:57,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-31 16:55:57,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-31 16:55:57,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-31 16:55:57,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-31 16:55:57,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-31 16:55:57,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-31 16:55:57,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-31 16:55:57,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-31 16:55:57,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-31 16:55:57,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-31 16:55:57,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-31 16:55:57,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-31 16:55:57,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-31 16:55:57,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-31 16:55:57,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-31 16:55:57,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-31 16:55:57,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-31 16:55:57,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-31 16:55:57,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-31 16:55:57,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-31 16:55:57,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-31 16:55:57,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-31 16:55:57,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-31 16:55:57,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-31 16:55:57,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-31 16:55:57,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-31 16:55:57,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-31 16:55:57,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-31 16:55:57,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-31 16:55:57,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-31 16:55:57,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-31 16:55:57,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-31 16:55:57,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-31 16:55:57,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-31 16:55:57,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-31 16:55:57,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-31 16:55:57,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-31 16:55:57,609 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019920655680937024); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03376820704388187, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019920655680937024); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:55:57,610 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-31 16:55:57,610 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014239746490973908, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019920655680937024); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03376820704388187, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019920655680937024); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-31 16:55:57,993 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 522 ms to try 5000 different iterations. Found a solution that moves 978 regions; Going from a computed imbalance of 0.014239746490973908 to a new imbalance of 0.001099197957032789. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005563406349598191); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01005037815259212); MoveCostFunction : (multiplier=7.0, imbalance=0.10866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005563406349598191); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:55:57,997 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1297953227:81 , srv1662268580:82 , srv168673422:82 , srv1843653218:82 , srv373779082:82 , srv1208890389:83 , srv1234176113:83 , srv1434779176:83 , srv1742176848:83 , srv1851885987:83 , srv560362471:83 , srv951213773:83 , srv1071458509:84 , srv1142933720:84 , srv1221386077:84 , srv1287846643:84 , srv1408368030:84 , srv1881958787:84 , srv2008119661:84 , srv2031148794:84 , srv266073327:84 , srv35367966:84 , srv472153332:84 , srv524208954:84 , srv641676810:84 , srv796940827:84 , srv813084598:84 , srv1041161441:85 , srv1117237659:85 , srv117777099:85 , srv1184606482:85 , srv1184941823:85 , srv147207403:85 , srv1523066739:85 , srv1597799559:85 , srv1686407390:85 , srv1688603373:85 , srv1828779618:85 , srv1848161103:85 , srv1871541119:85 , srv1913161006:85 , srv1965186536:85 , srv2063463830:85 , srv2096418030:85 , srv2123668190:85 , srv23039778:85 , srv230874629:85 , srv292744661:85 , srv360597693:85 , srv415664514:85 , srv579920910:85 , srv720080085:85 , srv800688241:85 , srv907348287:85 , srv95328671:85 , srv1029541511:86 , srv114398957:86 , srv1330916163:86 , srv1569330356:86 , srv1581020452:86 , srv1604877780:86 , srv1614521090:86 , srv1637063933:86 , srv1648022299:86 , srv1657527376:86 , srv1698511201:86 , srv1747394150:86 , srv1860722003:86 , srv1939209424:86 , srv1944846234:86 , srv345694099:86 , srv358085149:86 , srv398700446:86 , srv525443426:86 , srv537097432:86 , srv552168170:86 , srv728918563:86 , srv763034966:86 , srv828668929:86 , srv1234564289:87 , srv1723589500:87 , srv1797570544:87 , srv1853603965:87 , srv1951821118:87 , srv2032571174:87 , srv2132919177:87 , srv637340971:87 , srv675025885:87 , srv990166502:87 , srv994902771:87 , srv1203818828:88 , srv1930250262:88 , srv1995196997:88 , srv34927053:88 , srv358617637:88 , srv672030925:88 , srv826419682:88 , srv888336199:88 , srv192132763:89 , srv1018984439:556 } 2023-03-31 16:55:58,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 16:55:58,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv415664514=74, srv524208954=76, srv1208890389=11, srv1071458509=3, srv1234176113=13, srv230874629=63, srv1657527376=29, srv1853603965=43, srv2132919177=61, srv1637063933=27, srv1871541119=45, srv1434779176=19, srv1698511201=34, srv537097432=78, srv990166502=98, srv1029541511=1, srv907348287=95, srv1688603373=33, srv358617637=70, srv951213773=96, srv1184606482=8, srv1604877780=25, srv637340971=82, srv2031148794=56, srv35367966=68, srv1965186536=53, srv1747394150=37, srv168673422=32, srv292744661=65, srv360597693=71, srv1851885987=42, srv1944846234=51, srv1184941823=9, srv345694099=66, srv1041161441=2, srv1221386077=12, srv1297953227=16, srv1913161006=47, srv672030925=84, srv1287846643=15, srv192132763=48, srv2123668190=60, srv1686407390=31, srv1569330356=22, srv579920910=81, srv1860722003=44, srv373779082=72, srv1142933720=5, srv117777099=7, srv796940827=89, srv114398957=6, srv641676810=83, srv1723589500=35, srv1843653218=40, srv720080085=86, srv1995196997=54, srv1581020452=23, srv888336199=94, srv1203818828=10, srv1848161103=41, srv358085149=69, srv1408368030=18, srv1523066739=21, srv828668929=93, srv1662268580=30, srv1797570544=38, srv813084598=91, srv826419682=92, srv1828779618=39, srv2008119661=55, srv560362471=80, srv2032571174=57, srv552168170=79, srv472153332=75, srv2063463830=58, srv1742176848=36, srv994902771=99, srv34927053=67, srv1614521090=26, srv1117237659=4, srv1597799559=24, srv23039778=62, srv95328671=97, srv728918563=87, srv675025885=85, srv147207403=20, srv1234564289=14, srv398700446=73, srv1018984439=0, srv1330916163=17, srv2096418030=59, srv266073327=64, srv1881958787=46, srv1930250262=49, srv800688241=90, srv1648022299=28, srv525443426=77, srv763034966=88, srv1939209424=50, srv1951821118=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 16:55:58,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 16:55:58,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 16:55:58,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 16:55:58,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 16:55:58,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 16:55:58,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 16:55:58,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 16:55:58,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 16:55:58,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 16:55:58,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 16:55:58,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 16:55:58,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 16:55:58,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 16:55:58,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 16:55:58,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 16:55:58,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 16:55:58,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 16:55:58,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 16:55:58,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 16:55:58,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 16:55:58,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 16:55:58,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 16:55:58,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 16:55:58,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 16:55:58,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 16:55:58,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 16:55:58,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 16:55:58,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 16:55:58,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 16:55:58,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 16:55:58,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-31 16:55:58,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-31 16:55:58,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-31 16:55:58,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-31 16:55:58,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-31 16:55:58,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-31 16:55:58,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-31 16:55:58,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-31 16:55:58,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-31 16:55:58,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-31 16:55:58,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-31 16:55:58,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-31 16:55:58,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-31 16:55:58,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-31 16:55:58,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-31 16:55:58,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-31 16:55:58,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-31 16:55:58,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-31 16:55:58,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-31 16:55:58,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-31 16:55:58,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-31 16:55:58,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-31 16:55:58,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-31 16:55:58,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-31 16:55:58,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-31 16:55:58,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-31 16:55:58,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-31 16:55:58,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-31 16:55:58,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-31 16:55:58,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-31 16:55:58,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-31 16:55:58,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-31 16:55:58,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-31 16:55:58,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-31 16:55:58,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-31 16:55:58,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-31 16:55:58,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-31 16:55:58,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-31 16:55:58,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-31 16:55:58,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-31 16:55:58,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-31 16:55:58,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-31 16:55:58,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-31 16:55:58,188 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005563406349598191); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01005037815259212); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005563406349598191); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:55:58,188 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-31 16:55:58,188 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0010923498546653622, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005563406349598191); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01005037815259212); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005563406349598191); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-31 16:55:58,576 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 519 ms to try 5000 different iterations. Found a solution that moves 260 regions; Going from a computed imbalance of 0.0010923498546653622 to a new imbalance of 3.445043544085402E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.028888888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:55:58,577 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv34927053:82 , srv147207403:83 , srv1853603965:83 , srv1913161006:83 , srv1930250262:83 , srv1995196997:83 , srv672030925:83 , srv990166502:83 , srv1117237659:84 , srv114398957:84 , srv1203818828:84 , srv1234564289:84 , srv1330916163:84 , srv1569330356:84 , srv1581020452:84 , srv1597799559:84 , srv1614521090:84 , srv1637063933:84 , srv1657527376:84 , srv1662268580:84 , srv1686407390:84 , srv168673422:84 , srv1688603373:84 , srv1698511201:84 , srv1747394150:84 , srv1848161103:84 , srv192132763:84 , srv1939209424:84 , srv1951821118:84 , srv2032571174:84 , srv2123668190:84 , srv2132919177:84 , srv23039778:84 , srv230874629:84 , srv292744661:84 , srv358085149:84 , srv358617637:84 , srv398700446:84 , srv415664514:84 , srv524208954:84 , srv552168170:84 , srv560362471:84 , srv579920910:84 , srv637340971:84 , srv720080085:84 , srv763034966:84 , srv826419682:84 , srv828668929:84 , srv888336199:84 , srv994902771:84 , srv1029541511:85 , srv1071458509:85 , srv1142933720:85 , srv117777099:85 , srv1184941823:85 , srv1208890389:85 , srv1221386077:85 , srv1234176113:85 , srv1287846643:85 , srv1408368030:85 , srv1523066739:85 , srv1604877780:85 , srv1648022299:85 , srv1723589500:85 , srv1742176848:85 , srv1797570544:85 , srv1828779618:85 , srv1843653218:85 , srv1851885987:85 , srv1860722003:85 , srv1871541119:85 , srv1944846234:85 , srv1965186536:85 , srv2008119661:85 , srv2063463830:85 , srv2096418030:85 , srv266073327:85 , srv35367966:85 , srv360597693:85 , srv373779082:85 , srv472153332:85 , srv525443426:85 , srv537097432:85 , srv675025885:85 , srv728918563:85 , srv796940827:85 , srv813084598:85 , srv907348287:85 , srv951213773:85 , srv95328671:85 , srv1041161441:86 , srv1184606482:86 , srv1297953227:86 , srv1434779176:86 , srv1881958787:86 , srv2031148794:86 , srv345694099:86 , srv641676810:86 , srv800688241:86 , srv1018984439:635 } 2023-03-31 16:55:58,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 16:55:58,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv415664514=74, srv524208954=76, srv1208890389=11, srv1071458509=3, srv1234176113=13, srv230874629=63, srv1657527376=29, srv1853603965=43, srv2132919177=61, srv1637063933=27, srv1871541119=45, srv1434779176=19, srv1698511201=34, srv537097432=78, srv990166502=98, srv1029541511=1, srv907348287=95, srv1688603373=33, srv358617637=70, srv951213773=96, srv1184606482=8, srv1604877780=25, srv637340971=82, srv2031148794=56, srv35367966=68, srv1965186536=53, srv1747394150=37, srv168673422=32, srv292744661=65, srv360597693=71, srv1851885987=42, srv1944846234=51, srv1184941823=9, srv345694099=66, srv1041161441=2, srv1221386077=12, srv1297953227=16, srv1913161006=47, srv672030925=84, srv1287846643=15, srv192132763=48, srv2123668190=60, srv1686407390=31, srv1569330356=22, srv579920910=81, srv1860722003=44, srv373779082=72, srv1142933720=5, srv117777099=7, srv796940827=89, srv114398957=6, srv641676810=83, srv1723589500=35, srv1843653218=40, srv720080085=86, srv1995196997=54, srv1581020452=23, srv888336199=94, srv1203818828=10, srv1848161103=41, srv358085149=69, srv1408368030=18, srv1523066739=21, srv828668929=93, srv1662268580=30, srv1797570544=38, srv813084598=91, srv826419682=92, srv1828779618=39, srv2008119661=55, srv560362471=80, srv2032571174=57, srv552168170=79, srv472153332=75, srv2063463830=58, srv1742176848=36, srv994902771=99, srv34927053=67, srv1614521090=26, srv1117237659=4, srv1597799559=24, srv23039778=62, srv95328671=97, srv728918563=87, srv675025885=85, srv147207403=20, srv1234564289=14, srv398700446=73, srv1018984439=0, srv1330916163=17, srv2096418030=59, srv266073327=64, srv1881958787=46, srv1930250262=49, srv800688241=90, srv1648022299=28, srv525443426=77, srv763034966=88, srv1939209424=50, srv1951821118=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 16:55:58,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 16:55:58,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 16:55:58,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 16:55:58,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 16:55:58,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 16:55:58,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 16:55:58,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 16:55:58,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 16:55:58,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 16:55:58,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 16:55:58,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 16:55:58,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 16:55:58,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 16:55:58,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 16:55:58,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 16:55:58,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 16:55:58,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 16:55:58,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 16:55:58,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 16:55:58,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 16:55:58,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 16:55:58,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 16:55:58,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 16:55:58,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 16:55:58,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 16:55:58,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 16:55:58,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 16:55:58,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 16:55:58,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 16:55:58,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 16:55:58,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-31 16:55:58,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-31 16:55:58,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-31 16:55:58,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-31 16:55:58,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-31 16:55:58,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-31 16:55:58,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-31 16:55:58,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-31 16:55:58,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-31 16:55:58,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-31 16:55:58,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-31 16:55:58,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-31 16:55:58,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-31 16:55:58,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-31 16:55:58,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-31 16:55:58,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-31 16:55:58,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-31 16:55:58,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-31 16:55:58,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-31 16:55:58,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-31 16:55:58,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-31 16:55:58,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-31 16:55:58,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-31 16:55:58,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-31 16:55:58,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-31 16:55:58,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-31 16:55:58,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-31 16:55:58,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-31 16:55:58,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-31 16:55:58,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-31 16:55:58,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-31 16:55:58,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-31 16:55:58,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-31 16:55:58,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-31 16:55:58,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-31 16:55:58,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-31 16:55:58,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-31 16:55:58,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-31 16:55:58,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-31 16:55:58,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-31 16:55:58,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-31 16:55:58,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-31 16:55:58,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-31 16:55:58,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-31 16:55:58,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-31 16:55:58,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-31 16:55:58,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-31 16:55:58,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-31 16:55:58,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-31 16:55:58,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-31 16:55:58,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-31 16:55:58,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-31 16:55:58,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-31 16:55:58,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-31 16:55:58,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-31 16:55:58,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-31 16:55:58,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-31 16:55:58,721 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:55:58,721 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-31 16:55:58,721 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.2629876529268156E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-31 16:55:59,024 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 408 ms to try 5000 different iterations. Found a solution that moves 26 regions; Going from a computed imbalance of 3.2629876529268156E-5 to a new imbalance of 7.2552951874880335E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0028888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:55:59,025 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1408368030:83 , srv1871541119:83 , srv345694099:83 , srv796940827:83 , srv800688241:83 , srv95328671:83 , srv1029541511:84 , srv1041161441:84 , srv1071458509:84 , srv1117237659:84 , srv1142933720:84 , srv114398957:84 , srv117777099:84 , srv1184606482:84 , srv1184941823:84 , srv1203818828:84 , srv1208890389:84 , srv1221386077:84 , srv1234176113:84 , srv1234564289:84 , srv1287846643:84 , srv1297953227:84 , srv1330916163:84 , srv1434779176:84 , srv1523066739:84 , srv1569330356:84 , srv1581020452:84 , srv1597799559:84 , srv1604877780:84 , srv1614521090:84 , srv1637063933:84 , srv1648022299:84 , srv1657527376:84 , srv1662268580:84 , srv1686407390:84 , srv168673422:84 , srv1688603373:84 , srv1698511201:84 , srv1723589500:84 , srv1742176848:84 , srv1747394150:84 , srv1797570544:84 , srv1828779618:84 , srv1848161103:84 , srv1851885987:84 , srv1853603965:84 , srv1860722003:84 , srv1881958787:84 , srv192132763:84 , srv1930250262:84 , srv1939209424:84 , srv1944846234:84 , srv1951821118:84 , srv1965186536:84 , srv1995196997:84 , srv2008119661:84 , srv2031148794:84 , srv2032571174:84 , srv2063463830:84 , srv2096418030:84 , srv2132919177:84 , srv23039778:84 , srv266073327:84 , srv292744661:84 , srv34927053:84 , srv35367966:84 , srv358085149:84 , srv358617637:84 , srv360597693:84 , srv373779082:84 , srv398700446:84 , srv415664514:84 , srv472153332:84 , srv524208954:84 , srv525443426:84 , srv537097432:84 , srv552168170:84 , srv560362471:84 , srv579920910:84 , srv637340971:84 , srv641676810:84 , srv672030925:84 , srv675025885:84 , srv720080085:84 , srv728918563:84 , srv763034966:84 , srv813084598:84 , srv828668929:84 , srv888336199:84 , srv907348287:84 , srv951213773:84 , srv990166502:84 , srv994902771:84 , srv147207403:85 , srv1843653218:85 , srv1913161006:85 , srv2123668190:85 , srv230874629:85 , srv826419682:85 , srv1018984439:684 } 2023-03-31 16:55:59,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 16:55:59,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv415664514=74, srv524208954=76, srv1208890389=11, srv1071458509=3, srv1234176113=13, srv230874629=63, srv1657527376=29, srv1853603965=43, srv2132919177=61, srv1637063933=27, srv1871541119=45, srv1434779176=19, srv1698511201=34, srv537097432=78, srv990166502=98, srv1029541511=1, srv907348287=95, srv1688603373=33, srv358617637=70, srv951213773=96, srv1184606482=8, srv1604877780=25, srv637340971=82, srv2031148794=56, srv35367966=68, srv1965186536=53, srv1747394150=37, srv168673422=32, srv292744661=65, srv360597693=71, srv1851885987=42, srv1944846234=51, srv1184941823=9, srv345694099=66, srv1041161441=2, srv1221386077=12, srv1297953227=16, srv1913161006=47, srv672030925=84, srv1287846643=15, srv192132763=48, srv2123668190=60, srv1686407390=31, srv1569330356=22, srv579920910=81, srv1860722003=44, srv373779082=72, srv1142933720=5, srv117777099=7, srv796940827=89, srv114398957=6, srv641676810=83, srv1723589500=35, srv1843653218=40, srv720080085=86, srv1995196997=54, srv1581020452=23, srv888336199=94, srv1203818828=10, srv1848161103=41, srv358085149=69, srv1408368030=18, srv1523066739=21, srv828668929=93, srv1662268580=30, srv1797570544=38, srv813084598=91, srv826419682=92, srv1828779618=39, srv2008119661=55, srv560362471=80, srv2032571174=57, srv552168170=79, srv472153332=75, srv2063463830=58, srv1742176848=36, srv994902771=99, srv34927053=67, srv1614521090=26, srv1117237659=4, srv1597799559=24, srv23039778=62, srv95328671=97, srv728918563=87, srv675025885=85, srv147207403=20, srv1234564289=14, srv398700446=73, srv1018984439=0, srv1330916163=17, srv2096418030=59, srv266073327=64, srv1881958787=46, srv1930250262=49, srv800688241=90, srv1648022299=28, srv525443426=77, srv763034966=88, srv1939209424=50, srv1951821118=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 16:55:59,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 16:55:59,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 16:55:59,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 16:55:59,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 16:55:59,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 16:55:59,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 16:55:59,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 16:55:59,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 16:55:59,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 16:55:59,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 16:55:59,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 16:55:59,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 16:55:59,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 16:55:59,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 16:55:59,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 16:55:59,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 16:55:59,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 16:55:59,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 16:55:59,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 16:55:59,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 16:55:59,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 16:55:59,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 16:55:59,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 16:55:59,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 16:55:59,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 16:55:59,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 16:55:59,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 16:55:59,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 16:55:59,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 16:55:59,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 16:55:59,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-31 16:55:59,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-31 16:55:59,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-31 16:55:59,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-31 16:55:59,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-31 16:55:59,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-31 16:55:59,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-31 16:55:59,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-31 16:55:59,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-31 16:55:59,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-31 16:55:59,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-31 16:55:59,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-31 16:55:59,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-31 16:55:59,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-31 16:55:59,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-31 16:55:59,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-31 16:55:59,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-31 16:55:59,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-31 16:55:59,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-31 16:55:59,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-31 16:55:59,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-31 16:55:59,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-31 16:55:59,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-31 16:55:59,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-31 16:55:59,174 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:55:59,175 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1408368030:83 , srv1871541119:83 , srv345694099:83 , srv796940827:83 , srv800688241:83 , srv95328671:83 , srv1029541511:84 , srv1041161441:84 , srv1071458509:84 , srv1117237659:84 , srv1142933720:84 , srv114398957:84 , srv117777099:84 , srv1184606482:84 , srv1184941823:84 , srv1203818828:84 , srv1208890389:84 , srv1221386077:84 , srv1234176113:84 , srv1234564289:84 , srv1287846643:84 , srv1297953227:84 , srv1330916163:84 , srv1434779176:84 , srv1523066739:84 , srv1569330356:84 , srv1581020452:84 , srv1597799559:84 , srv1604877780:84 , srv1614521090:84 , srv1637063933:84 , srv1648022299:84 , srv1657527376:84 , srv1662268580:84 , srv1686407390:84 , srv168673422:84 , srv1688603373:84 , srv1698511201:84 , srv1723589500:84 , srv1742176848:84 , srv1747394150:84 , srv1797570544:84 , srv1828779618:84 , srv1848161103:84 , srv1851885987:84 , srv1853603965:84 , srv1860722003:84 , srv1881958787:84 , srv192132763:84 , srv1930250262:84 , srv1939209424:84 , srv1944846234:84 , srv1951821118:84 , srv1965186536:84 , srv1995196997:84 , srv2008119661:84 , srv2031148794:84 , srv2032571174:84 , srv2063463830:84 , srv2096418030:84 , srv2132919177:84 , srv23039778:84 , srv266073327:84 , srv292744661:84 , srv34927053:84 , srv35367966:84 , srv358085149:84 , srv358617637:84 , srv360597693:84 , srv373779082:84 , srv398700446:84 , srv415664514:84 , srv472153332:84 , srv524208954:84 , srv525443426:84 , srv537097432:84 , srv552168170:84 , srv560362471:84 , srv579920910:84 , srv637340971:84 , srv641676810:84 , srv672030925:84 , srv675025885:84 , srv720080085:84 , srv728918563:84 , srv763034966:84 , srv813084598:84 , srv828668929:84 , srv888336199:84 , srv907348287:84 , srv951213773:84 , srv990166502:84 , srv994902771:84 , srv147207403:85 , srv1843653218:85 , srv1913161006:85 , srv2123668190:85 , srv230874629:85 , srv826419682:85 , srv1018984439:684 } 2023-03-31 16:55:59,203 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=458 (was 454) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=4019 (was 4508) 2023-03-31 16:55:59,212 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=458, ProcessCount=169, AvailableMemoryMB=4019 2023-03-31 16:55:59,213 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-31 16:55:59,213 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-03-31 16:55:59,223 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1124456666:84 , srv1181432398:84 , srv1190849387:84 , srv1303743104:84 , srv1453363227:84 , srv1482464155:84 , srv1567784987:84 , srv1687691031:84 , srv1702120381:84 , srv1706774804:84 , srv1744930145:84 , srv1773386231:84 , srv1834860645:84 , srv1852672028:84 , srv1855948605:84 , srv1939720169:84 , srv1983735031:84 , srv2002269120:84 , srv2070890283:84 , srv2114731263:84 , srv239292363:84 , srv42710521:84 , srv44409501:84 , srv551140954:84 , srv704547107:84 , srv712761033:84 , srv875696032:84 , srv888876604:84 , srv944161650:84 , srv1280626763:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-03-31 16:55:59,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 16:55:59,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1852672028=14, srv42710521=22, srv239292363=21, srv1181432398=1, srv1124456666=0, srv1453363227=5, srv1567784987=7, srv1939720169=16, srv1190849387=2, srv1773386231=12, srv1983735031=17, srv888876604=28, srv2114731263=20, srv1303743104=4, srv1687691031=8, srv1834860645=13, srv44409501=23, srv1706774804=10, srv944161650=29, srv1482464155=6, srv2070890283=19, srv551140954=24, srv1280626763=3, srv1855948605=15, srv712761033=26, srv704547107=25, srv1702120381=9, srv2002269120=18, srv1744930145=11, srv875696032=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 16:55:59,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 16:55:59,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 16:55:59,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 16:55:59,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 16:55:59,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 16:55:59,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 16:55:59,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 16:55:59,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 16:55:59,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 16:55:59,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 16:55:59,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 16:55:59,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 16:55:59,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 16:55:59,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 16:55:59,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 16:55:59,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 16:55:59,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-31 16:55:59,266 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0780851935249658, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:55:59,267 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912439087159362, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0780851935249658, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-03-31 16:57:59,245 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6146009 different iterations. Found a solution that moves 1859 regions; Going from a computed imbalance of 0.9912439087159362 to a new imbalance of 4.338998739279625E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6885185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:57:59,253 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1124456666:90 , srv1181432398:90 , srv1190849387:90 , srv1280626763:90 , srv1303743104:90 , srv1453363227:90 , srv1482464155:90 , srv1567784987:90 , srv1687691031:90 , srv1702120381:90 , srv1706774804:90 , srv1744930145:90 , srv1773386231:90 , srv1834860645:90 , srv1852672028:90 , srv1855948605:90 , srv1939720169:90 , srv1983735031:90 , srv2002269120:90 , srv2070890283:90 , srv2114731263:90 , srv239292363:90 , srv42710521:90 , srv44409501:90 , srv551140954:90 , srv704547107:90 , srv712761033:90 , srv875696032:90 , srv888876604:90 , srv944161650:90 } 2023-03-31 16:57:59,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 16:57:59,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1852672028=14, srv42710521=22, srv239292363=21, srv1181432398=1, srv1124456666=0, srv1453363227=5, srv1567784987=7, srv1939720169=16, srv1190849387=2, srv1773386231=12, srv1983735031=17, srv888876604=28, srv2114731263=20, srv1303743104=4, srv1687691031=8, srv1834860645=13, srv44409501=23, srv1706774804=10, srv944161650=29, srv1482464155=6, srv2070890283=19, srv551140954=24, srv1280626763=3, srv1855948605=15, srv712761033=26, srv704547107=25, srv1702120381=9, srv2002269120=18, srv1744930145=11, srv875696032=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 16:57:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 16:57:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 16:57:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 16:57:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 16:57:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 16:57:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 16:57:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 16:57:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 16:57:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 16:57:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 16:57:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 16:57:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 16:57:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 16:57:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-31 16:57:59,271 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 16:57:59,271 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1124456666:90 , srv1181432398:90 , srv1190849387:90 , srv1280626763:90 , srv1303743104:90 , srv1453363227:90 , srv1482464155:90 , srv1567784987:90 , srv1687691031:90 , srv1702120381:90 , srv1706774804:90 , srv1744930145:90 , srv1773386231:90 , srv1834860645:90 , srv1852672028:90 , srv1855948605:90 , srv1939720169:90 , srv1983735031:90 , srv2002269120:90 , srv2070890283:90 , srv2114731263:90 , srv239292363:90 , srv42710521:90 , srv44409501:90 , srv551140954:90 , srv704547107:90 , srv712761033:90 , srv875696032:90 , srv888876604:90 , srv944161650:90 } 2023-03-31 16:57:59,284 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=260 (was 458), ProcessCount=169 (was 169), AvailableMemoryMB=3923 (was 4019)