2023-10-12 21:57:05,644 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-12 21:57:05,892 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-12 21:57:05,982 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-12 21:57:06,039 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-12 21:57:06,048 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-12 21:57:06,060 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=311, ProcessCount=167, AvailableMemoryMB=9243 2023-10-12 21:57:06,062 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-12 21:57:06,062 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-12 21:57:06,182 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1003139334:84 , srv105924456:84 , srv1064965334:84 , srv108714086:84 , srv1090247101:84 , srv1181079001:84 , srv1191015072:84 , srv1197979915:84 , srv1282846454:84 , srv1286163371:84 , srv1288552317:84 , srv1311606431:84 , srv1311638490:84 , srv1321823423:84 , srv1352891095:84 , srv139232624:84 , srv1394154233:84 , srv1432740081:84 , srv1451518071:84 , srv146861537:84 , srv1474413758:84 , srv148433082:84 , srv1518250624:84 , srv1526861986:84 , srv1566462628:84 , srv1592106032:84 , srv1599100141:84 , srv1619693807:84 , srv1622861453:84 , srv1659880464:84 , srv166168422:84 , srv1683287280:84 , srv1698761603:84 , srv1721206675:84 , srv1726097686:84 , srv1781234293:84 , srv1792503217:84 , srv1794452657:84 , srv1804224077:84 , srv1805965488:84 , srv1827385525:84 , srv1833976775:84 , srv1881211681:84 , srv1890525517:84 , srv189258586:84 , srv1899963456:84 , srv190775053:84 , srv1931225420:84 , srv1963069380:84 , srv1966145979:84 , srv1989757419:84 , srv1990107384:84 , srv2062869808:84 , srv2095120020:84 , srv2095796285:84 , srv2103058891:84 , srv26112956:84 , srv270445798:84 , srv29755771:84 , srv323789886:84 , srv332058768:84 , srv368497152:84 , srv382824755:84 , srv412818934:84 , srv425323312:84 , srv439452672:84 , srv444596947:84 , srv456440658:84 , srv456907836:84 , srv512992635:84 , srv523706886:84 , srv536202956:84 , srv542025396:84 , srv569247299:84 , srv580839121:84 , srv599451029:84 , srv608334095:84 , srv614113403:84 , srv642572020:84 , srv651691339:84 , srv687192818:84 , srv771234423:84 , srv771266679:84 , srv787363269:84 , srv798283497:84 , srv831364257:84 , srv851478610:84 , srv856169262:84 , srv863300430:84 , srv887869959:84 , srv892277310:84 , srv894016511:84 , srv926692834:84 , srv938045976:84 , srv942932168:84 , srv948522465:84 , srv959054195:84 , srv978760109:84 , srv997386695:84 , srv1446255634:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-12 21:57:06,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 21:57:06,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2103058891=56, srv1599100141=27, srv1526861986=24, srv997386695=99, srv456907836=69, srv978760109=98, srv1090247101=4, srv1966145979=50, srv1890525517=44, srv1446255634=18, srv1451518071=19, srv523706886=71, srv771234423=82, srv332058768=61, srv1282846454=8, srv146861537=20, srv787363269=84, srv599451029=76, srv1394154233=16, srv959054195=97, srv382824755=63, srv1321823423=13, srv1432740081=17, srv2062869808=53, srv892277310=91, srv1352891095=14, srv1989757419=51, srv444596947=67, srv851478610=87, srv948522465=96, srv1931225420=48, srv1566462628=25, srv2095796285=55, srv323789886=60, srv439452672=66, srv580839121=75, srv771266679=83, srv425323312=65, srv856169262=88, srv26112956=57, srv608334095=77, srv1064965334=2, srv1619693807=28, srv1804224077=39, srv887869959=90, srv614113403=78, srv926692834=93, srv1721206675=34, srv1474413758=21, srv542025396=73, srv512992635=70, srv105924456=1, srv536202956=72, srv1311638490=12, srv1181079001=5, srv642572020=79, srv687192818=81, srv894016511=92, srv1726097686=35, srv938045976=94, srv2095120020=54, srv1827385525=41, srv108714086=3, srv139232624=15, srv1518250624=23, srv190775053=47, srv569247299=74, srv368497152=62, srv189258586=45, srv798283497=85, srv1833976775=42, srv1659880464=30, srv1805965488=40, srv29755771=59, srv1698761603=33, srv863300430=89, srv412818934=64, srv1003139334=0, srv1191015072=6, srv1794452657=38, srv1197979915=7, srv1781234293=36, srv148433082=22, srv942932168=95, srv166168422=31, srv1881211681=43, srv456440658=68, srv1622861453=29, srv1311606431=11, srv270445798=58, srv1683287280=32, srv1963069380=49, srv651691339=80, srv1899963456=46, srv1792503217=37, srv1990107384=52, srv1286163371=9, srv1592106032=26, srv1288552317=10, srv831364257=86} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 21:57:06,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 21:57:06,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 21:57:06,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 21:57:06,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 21:57:06,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 21:57:06,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 21:57:06,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 21:57:06,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 21:57:06,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 21:57:06,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 21:57:06,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 21:57:06,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 21:57:06,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 21:57:06,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 21:57:06,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 21:57:06,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 21:57:06,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 21:57:06,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 21:57:06,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 21:57:06,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 21:57:06,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 21:57:06,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 21:57:06,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 21:57:06,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 21:57:06,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 21:57:06,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 21:57:06,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 21:57:06,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 21:57:06,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 21:57:06,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 21:57:06,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-12 21:57:06,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-12 21:57:06,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-12 21:57:06,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-12 21:57:06,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-12 21:57:06,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-12 21:57:06,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-12 21:57:06,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-12 21:57:06,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-12 21:57:06,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-12 21:57:06,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-12 21:57:06,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-12 21:57:06,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-12 21:57:06,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-12 21:57:06,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-12 21:57:06,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-12 21:57:06,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-12 21:57:06,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-12 21:57:06,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-12 21:57:06,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-12 21:57:06,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-12 21:57:06,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-12 21:57:06,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-12 21:57:06,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-12 21:57:06,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-12 21:57:06,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-12 21:57:06,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-12 21:57:06,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-12 21:57:06,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-12 21:57:06,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-12 21:57:06,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-12 21:57:06,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-12 21:57:06,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-12 21:57:06,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-12 21:57:06,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-12 21:57:06,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-12 21:57:06,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-12 21:57:06,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-12 21:57:06,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-12 21:57:06,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-12 21:57:06,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-12 21:57:06,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-12 21:57:06,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-12 21:57:06,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-12 21:57:06,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-12 21:57:06,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-12 21:57:06,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-12 21:57:06,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-12 21:57:06,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-12 21:57:06,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-12 21:57:06,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-12 21:57:06,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-12 21:57:06,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-12 21:57:06,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-12 21:57:06,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-12 21:57:06,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-12 21:57:06,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-12 21:57:06,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-12 21:57:06,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-12 21:57:06,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-12 21:57:06,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-12 21:57:06,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-12 21:57:06,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-12 21:57:06,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-12 21:57:06,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-12 21:57:06,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-12 21:57:06,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-12 21:57:06,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-12 21:57:06,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-12 21:57:06,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-12 21:57:06,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-12 21:57:06,413 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 21:57:06,414 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-12 21:57:06,414 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-12 21:57:06,823 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 531 ms to try 5000 different iterations. Found a solution that moves 4661 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18574284489160642. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04018848223064654, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053263843618697265, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5178888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04018848223064654, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18125, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 21:57:06,833 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv105924456:81 , srv608334095:81 , srv1394154233:82 , srv1781234293:82 , srv1792503217:82 , srv1804224077:82 , srv580839121:82 , srv851478610:82 , srv1181079001:83 , srv1282846454:83 , srv1432740081:83 , srv1451518071:83 , srv1566462628:83 , srv1726097686:83 , srv1899963456:83 , srv425323312:83 , srv536202956:83 , srv1288552317:84 , srv1311606431:84 , srv1311638490:84 , srv1683287280:84 , srv542025396:84 , srv642572020:84 , srv651691339:84 , srv894016511:84 , srv948522465:84 , srv959054195:84 , srv108714086:85 , srv1321823423:85 , srv1622861453:85 , srv1698761603:85 , srv2103058891:85 , srv26112956:85 , srv270445798:85 , srv887869959:85 , srv926692834:85 , srv978760109:85 , srv997386695:85 , srv1191015072:86 , srv148433082:86 , srv1599100141:86 , srv1890525517:86 , srv1989757419:86 , srv2095796285:86 , srv439452672:86 , srv456907836:86 , srv512992635:86 , srv523706886:86 , srv599451029:86 , srv614113403:86 , srv787363269:86 , srv798283497:86 , srv856169262:86 , srv863300430:86 , srv1286163371:87 , srv146861537:87 , srv166168422:87 , srv1721206675:87 , srv1827385525:87 , srv1833976775:87 , srv1990107384:87 , srv29755771:87 , srv323789886:87 , srv332058768:87 , srv938045976:87 , srv942932168:87 , srv1003139334:88 , srv1518250624:88 , srv189258586:88 , srv190775053:88 , srv368497152:88 , srv382824755:88 , srv456440658:88 , srv687192818:88 , srv892277310:88 , srv1352891095:89 , srv139232624:89 , srv1474413758:89 , srv1805965488:89 , srv1931225420:89 , srv2062869808:89 , srv2095120020:89 , srv831364257:89 , srv1090247101:90 , srv1197979915:90 , srv1619693807:90 , srv1659880464:90 , srv1963069380:90 , srv1966145979:90 , srv444596947:90 , srv569247299:90 , srv771234423:90 , srv1526861986:91 , srv1794452657:91 , srv412818934:91 , srv771266679:91 , srv1592106032:92 , srv1881211681:92 , srv1064965334:95 , srv1446255634:447 } 2023-10-12 21:57:06,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 21:57:06,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2103058891=56, srv1599100141=27, srv1526861986=24, srv997386695=99, srv456907836=69, srv978760109=98, srv1090247101=4, srv1966145979=50, srv1890525517=44, srv1446255634=18, srv1451518071=19, srv523706886=71, srv771234423=82, srv332058768=61, srv1282846454=8, srv146861537=20, srv787363269=84, srv599451029=76, srv1394154233=16, srv959054195=97, srv382824755=63, srv1321823423=13, srv1432740081=17, srv2062869808=53, srv892277310=91, srv1352891095=14, srv1989757419=51, srv444596947=67, srv851478610=87, srv948522465=96, srv1931225420=48, srv1566462628=25, srv2095796285=55, srv323789886=60, srv439452672=66, srv580839121=75, srv771266679=83, srv425323312=65, srv856169262=88, srv26112956=57, srv608334095=77, srv1064965334=2, srv1619693807=28, srv1804224077=39, srv887869959=90, srv614113403=78, srv926692834=93, srv1721206675=34, srv1474413758=21, srv542025396=73, srv512992635=70, srv105924456=1, srv536202956=72, srv1311638490=12, srv1181079001=5, srv642572020=79, srv687192818=81, srv894016511=92, srv1726097686=35, srv938045976=94, srv2095120020=54, srv1827385525=41, srv108714086=3, srv139232624=15, srv1518250624=23, srv190775053=47, srv569247299=74, srv368497152=62, srv189258586=45, srv798283497=85, srv1833976775=42, srv1659880464=30, srv1805965488=40, srv29755771=59, srv1698761603=33, srv863300430=89, srv412818934=64, srv1003139334=0, srv1191015072=6, srv1794452657=38, srv1197979915=7, srv1781234293=36, srv148433082=22, srv942932168=95, srv166168422=31, srv1881211681=43, srv456440658=68, srv1622861453=29, srv1311606431=11, srv270445798=58, srv1683287280=32, srv1963069380=49, srv651691339=80, srv1899963456=46, srv1792503217=37, srv1990107384=52, srv1286163371=9, srv1592106032=26, srv1288552317=10, srv831364257=86} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 21:57:06,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 21:57:06,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 21:57:06,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 21:57:06,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 21:57:06,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 21:57:06,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 21:57:06,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 21:57:06,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 21:57:06,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 21:57:06,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 21:57:06,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 21:57:06,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 21:57:06,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 21:57:06,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 21:57:06,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 21:57:06,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 21:57:06,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 21:57:06,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 21:57:06,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 21:57:06,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 21:57:06,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 21:57:06,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 21:57:06,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 21:57:06,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 21:57:06,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 21:57:06,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 21:57:06,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 21:57:06,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 21:57:06,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 21:57:06,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 21:57:06,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-12 21:57:06,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-12 21:57:06,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-12 21:57:06,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-12 21:57:06,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-12 21:57:06,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-12 21:57:06,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-12 21:57:06,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-12 21:57:06,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-12 21:57:06,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-12 21:57:06,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-12 21:57:06,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-12 21:57:06,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-12 21:57:06,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-12 21:57:06,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-12 21:57:06,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-12 21:57:06,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-12 21:57:06,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-12 21:57:06,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-12 21:57:06,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-12 21:57:06,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-12 21:57:06,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-12 21:57:06,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-12 21:57:06,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-12 21:57:06,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-12 21:57:06,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-12 21:57:06,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-12 21:57:06,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-12 21:57:06,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-12 21:57:06,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-12 21:57:06,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-12 21:57:06,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-12 21:57:06,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-12 21:57:06,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-12 21:57:06,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-12 21:57:06,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-12 21:57:06,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-12 21:57:06,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-12 21:57:06,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-12 21:57:06,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-12 21:57:06,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-12 21:57:06,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-12 21:57:06,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-12 21:57:06,985 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04018848223064654, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053263843618697265, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04018848223064654, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18125, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 21:57:06,986 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-12 21:57:06,986 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18571020787204143, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04018848223064654, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053263843618697265, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04018848223064654, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18125, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-12 21:57:07,171 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 283 ms to try 5000 different iterations. Found a solution that moves 2617 regions; Going from a computed imbalance of 0.18571020787204143 to a new imbalance of 0.01658551074961487. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020152183196665542); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.027237163493954514, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2907777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020152183196665542); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 21:57:07,180 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1064965334:78 , srv1592106032:78 , srv1526861986:81 , srv1659880464:81 , srv1794452657:81 , srv1881211681:81 , srv412818934:81 , srv444596947:81 , srv1352891095:82 , srv1619693807:82 , srv1805965488:82 , srv569247299:82 , srv1003139334:83 , srv1090247101:83 , srv1197979915:83 , srv1518250624:83 , srv1827385525:83 , srv1833976775:83 , srv190775053:83 , srv1963069380:83 , srv1966145979:83 , srv1990107384:83 , srv2095120020:83 , srv687192818:83 , srv771234423:83 , srv831364257:83 , srv1721206675:84 , srv189258586:84 , srv2095796285:84 , srv368497152:84 , srv456440658:84 , srv771266679:84 , srv892277310:84 , srv1321823423:85 , srv139232624:85 , srv146861537:85 , srv1474413758:85 , srv1599100141:85 , srv1890525517:85 , srv1931225420:85 , srv1989757419:85 , srv2062869808:85 , srv26112956:85 , srv323789886:85 , srv599451029:85 , srv1191015072:86 , srv270445798:86 , srv29755771:86 , srv332058768:86 , srv439452672:86 , srv456907836:86 , srv523706886:86 , srv614113403:86 , srv651691339:86 , srv887869959:86 , srv938045976:86 , srv942932168:86 , srv997386695:86 , srv1288552317:87 , srv1311606431:87 , srv148433082:87 , srv382824755:87 , srv512992635:87 , srv863300430:87 , srv926692834:87 , srv959054195:87 , srv108714086:88 , srv1282846454:88 , srv1286163371:88 , srv1566462628:88 , srv1622861453:88 , srv1683287280:88 , srv1698761603:88 , srv1804224077:88 , srv425323312:88 , srv542025396:88 , srv798283497:88 , srv856169262:88 , srv948522465:88 , srv978760109:88 , srv1181079001:89 , srv1311638490:89 , srv1432740081:89 , srv1451518071:89 , srv1726097686:89 , srv1781234293:89 , srv1899963456:89 , srv2103058891:89 , srv642572020:89 , srv787363269:89 , srv105924456:90 , srv1394154233:90 , srv851478610:90 , srv894016511:90 , srv166168422:91 , srv1792503217:91 , srv608334095:91 , srv580839121:92 , srv536202956:93 , srv1446255634:506 } 2023-10-12 21:57:07,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 21:57:07,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2103058891=56, srv1599100141=27, srv1526861986=24, srv997386695=99, srv456907836=69, srv978760109=98, srv1090247101=4, srv1966145979=50, srv1890525517=44, srv1446255634=18, srv1451518071=19, srv523706886=71, srv771234423=82, srv332058768=61, srv1282846454=8, srv146861537=20, srv787363269=84, srv599451029=76, srv1394154233=16, srv959054195=97, srv382824755=63, srv1321823423=13, srv1432740081=17, srv2062869808=53, srv892277310=91, srv1352891095=14, srv1989757419=51, srv444596947=67, srv851478610=87, srv948522465=96, srv1931225420=48, srv1566462628=25, srv2095796285=55, srv323789886=60, srv439452672=66, srv580839121=75, srv771266679=83, srv425323312=65, srv856169262=88, srv26112956=57, srv608334095=77, srv1064965334=2, srv1619693807=28, srv1804224077=39, srv887869959=90, srv614113403=78, srv926692834=93, srv1721206675=34, srv1474413758=21, srv542025396=73, srv512992635=70, srv105924456=1, srv536202956=72, srv1311638490=12, srv1181079001=5, srv642572020=79, srv687192818=81, srv894016511=92, srv1726097686=35, srv938045976=94, srv2095120020=54, srv1827385525=41, srv108714086=3, srv139232624=15, srv1518250624=23, srv190775053=47, srv569247299=74, srv368497152=62, srv189258586=45, srv798283497=85, srv1833976775=42, srv1659880464=30, srv1805965488=40, srv29755771=59, srv1698761603=33, srv863300430=89, srv412818934=64, srv1003139334=0, srv1191015072=6, srv1794452657=38, srv1197979915=7, srv1781234293=36, srv148433082=22, srv942932168=95, srv166168422=31, srv1881211681=43, srv456440658=68, srv1622861453=29, srv1311606431=11, srv270445798=58, srv1683287280=32, srv1963069380=49, srv651691339=80, srv1899963456=46, srv1792503217=37, srv1990107384=52, srv1286163371=9, srv1592106032=26, srv1288552317=10, srv831364257=86} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 21:57:07,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 21:57:07,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 21:57:07,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-12 21:57:07,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-12 21:57:07,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-12 21:57:07,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-12 21:57:07,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-12 21:57:07,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-12 21:57:07,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-12 21:57:07,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-12 21:57:07,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-12 21:57:07,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-12 21:57:07,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-12 21:57:07,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-12 21:57:07,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-12 21:57:07,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-12 21:57:07,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-12 21:57:07,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-12 21:57:07,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-12 21:57:07,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-12 21:57:07,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-12 21:57:07,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-12 21:57:07,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-12 21:57:07,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-12 21:57:07,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-12 21:57:07,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-12 21:57:07,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-12 21:57:07,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-12 21:57:07,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-12 21:57:07,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-12 21:57:07,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-12 21:57:07,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-12 21:57:07,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-12 21:57:07,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-12 21:57:07,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-12 21:57:07,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-12 21:57:07,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-12 21:57:07,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-12 21:57:07,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-12 21:57:07,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-12 21:57:07,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-12 21:57:07,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-12 21:57:07,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-12 21:57:07,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-12 21:57:07,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-12 21:57:07,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-12 21:57:07,305 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020152183196665542); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.027237163493954514, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020152183196665542); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 21:57:07,305 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-12 21:57:07,305 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016567186123954795, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020152183196665542); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.027237163493954514, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020152183196665542); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-12 21:57:07,456 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 224 ms to try 5000 different iterations. Found a solution that moves 984 regions; Going from a computed imbalance of 0.016567186123954795 to a new imbalance of 6.018590535992679E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0050929751922394705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005058579404262188); MoveCostFunction : (multiplier=7.0, imbalance=0.10933333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0050929751922394705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 21:57:07,457 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv166168422:80 , srv536202956:82 , srv1286163371:83 , srv1474413758:83 , srv382824755:83 , srv580839121:83 , srv771266679:83 , srv787363269:83 , srv1064965334:84 , srv1311638490:84 , srv139232624:84 , srv148433082:84 , srv1622861453:84 , srv1792503217:84 , srv1881211681:84 , srv1963069380:84 , srv1966145979:84 , srv2103058891:84 , srv856169262:84 , srv863300430:84 , srv894016511:84 , srv942932168:84 , srv978760109:84 , srv108714086:85 , srv1090247101:85 , srv1191015072:85 , srv1197979915:85 , srv1432740081:85 , srv1526861986:85 , srv1619693807:85 , srv1683287280:85 , srv1698761603:85 , srv1726097686:85 , srv1794452657:85 , srv1931225420:85 , srv2062869808:85 , srv29755771:85 , srv332058768:85 , srv368497152:85 , srv439452672:85 , srv512992635:85 , srv542025396:85 , srv608334095:85 , srv642572020:85 , srv771234423:85 , srv798283497:85 , srv831364257:85 , srv851478610:85 , srv892277310:85 , srv938045976:85 , srv948522465:85 , srv1003139334:86 , srv105924456:86 , srv1181079001:86 , srv1288552317:86 , srv1352891095:86 , srv1394154233:86 , srv1451518071:86 , srv1518250624:86 , srv1566462628:86 , srv1599100141:86 , srv1659880464:86 , srv189258586:86 , srv1899963456:86 , srv1989757419:86 , srv2095120020:86 , srv270445798:86 , srv323789886:86 , srv412818934:86 , srv425323312:86 , srv444596947:86 , srv456440658:86 , srv456907836:86 , srv523706886:86 , srv569247299:86 , srv599451029:86 , srv614113403:86 , srv687192818:86 , srv887869959:86 , srv926692834:86 , srv959054195:86 , srv997386695:86 , srv1282846454:87 , srv1311606431:87 , srv1321823423:87 , srv146861537:87 , srv1592106032:87 , srv1721206675:87 , srv1805965488:87 , srv1827385525:87 , srv1833976775:87 , srv1890525517:87 , srv190775053:87 , srv2095796285:87 , srv1781234293:88 , srv1804224077:88 , srv1990107384:88 , srv26112956:88 , srv651691339:88 , srv1446255634:550 } 2023-10-12 21:57:07,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 21:57:07,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2103058891=56, srv1599100141=27, srv1526861986=24, srv997386695=99, srv456907836=69, srv978760109=98, srv1090247101=4, srv1966145979=50, srv1890525517=44, srv1446255634=18, srv1451518071=19, srv523706886=71, srv771234423=82, srv332058768=61, srv1282846454=8, srv146861537=20, srv787363269=84, srv599451029=76, srv1394154233=16, srv959054195=97, srv382824755=63, srv1321823423=13, srv1432740081=17, srv2062869808=53, srv892277310=91, srv1352891095=14, srv1989757419=51, srv444596947=67, srv851478610=87, srv948522465=96, srv1931225420=48, srv1566462628=25, srv2095796285=55, srv323789886=60, srv439452672=66, srv580839121=75, srv771266679=83, srv425323312=65, srv856169262=88, srv26112956=57, srv608334095=77, srv1064965334=2, srv1619693807=28, srv1804224077=39, srv887869959=90, srv614113403=78, srv926692834=93, srv1721206675=34, srv1474413758=21, srv542025396=73, srv512992635=70, srv105924456=1, srv536202956=72, srv1311638490=12, srv1181079001=5, srv642572020=79, srv687192818=81, srv894016511=92, srv1726097686=35, srv938045976=94, srv2095120020=54, srv1827385525=41, srv108714086=3, srv139232624=15, srv1518250624=23, srv190775053=47, srv569247299=74, srv368497152=62, srv189258586=45, srv798283497=85, srv1833976775=42, srv1659880464=30, srv1805965488=40, srv29755771=59, srv1698761603=33, srv863300430=89, srv412818934=64, srv1003139334=0, srv1191015072=6, srv1794452657=38, srv1197979915=7, srv1781234293=36, srv148433082=22, srv942932168=95, srv166168422=31, srv1881211681=43, srv456440658=68, srv1622861453=29, srv1311606431=11, srv270445798=58, srv1683287280=32, srv1963069380=49, srv651691339=80, srv1899963456=46, srv1792503217=37, srv1990107384=52, srv1286163371=9, srv1592106032=26, srv1288552317=10, srv831364257=86} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 21:57:07,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-12 21:57:07,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-12 21:57:07,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-12 21:57:07,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-12 21:57:07,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-12 21:57:07,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-12 21:57:07,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-12 21:57:07,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-12 21:57:07,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-12 21:57:07,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-12 21:57:07,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-12 21:57:07,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-12 21:57:07,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-12 21:57:07,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-12 21:57:07,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-12 21:57:07,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-12 21:57:07,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-12 21:57:07,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-12 21:57:07,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-12 21:57:07,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-12 21:57:07,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-12 21:57:07,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-12 21:57:07,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-12 21:57:07,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-12 21:57:07,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-12 21:57:07,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-12 21:57:07,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-12 21:57:07,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-12 21:57:07,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-12 21:57:07,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-12 21:57:07,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-12 21:57:07,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-12 21:57:07,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-12 21:57:07,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-12 21:57:07,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-12 21:57:07,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-12 21:57:07,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-12 21:57:07,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-12 21:57:07,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-12 21:57:07,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-12 21:57:07,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-12 21:57:07,553 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0050929751922394705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005058579404262188); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0050929751922394705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 21:57:07,553 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-12 21:57:07,553 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.949689383338815E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0050929751922394705); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005058579404262188); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0050929751922394705); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-12 21:57:07,775 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 292 ms to try 5000 different iterations. Found a solution that moves 231 regions; Going from a computed imbalance of 5.949689383338815E-4 to a new imbalance of 1.7825040699733642E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.025666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 21:57:07,776 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv146861537:83 , srv1781234293:83 , srv1931225420:83 , srv2062869808:83 , srv332058768:83 , srv798283497:83 , srv856169262:83 , srv108714086:84 , srv1090247101:84 , srv1181079001:84 , srv1282846454:84 , srv1286163371:84 , srv1311606431:84 , srv139232624:84 , srv1394154233:84 , srv1451518071:84 , srv1659880464:84 , srv166168422:84 , srv1698761603:84 , srv1721206675:84 , srv1804224077:84 , srv1890525517:84 , srv189258586:84 , srv1899963456:84 , srv190775053:84 , srv1990107384:84 , srv2095120020:84 , srv2103058891:84 , srv26112956:84 , srv29755771:84 , srv323789886:84 , srv382824755:84 , srv412818934:84 , srv456440658:84 , srv456907836:84 , srv512992635:84 , srv523706886:84 , srv536202956:84 , srv569247299:84 , srv614113403:84 , srv642572020:84 , srv651691339:84 , srv771234423:84 , srv771266679:84 , srv787363269:84 , srv894016511:84 , srv926692834:84 , srv938045976:84 , srv1003139334:85 , srv105924456:85 , srv1064965334:85 , srv1191015072:85 , srv1197979915:85 , srv1288552317:85 , srv1311638490:85 , srv1321823423:85 , srv1352891095:85 , srv1432740081:85 , srv1474413758:85 , srv148433082:85 , srv1518250624:85 , srv1526861986:85 , srv1566462628:85 , srv1592106032:85 , srv1599100141:85 , srv1619693807:85 , srv1622861453:85 , srv1683287280:85 , srv1726097686:85 , srv1792503217:85 , srv1794452657:85 , srv1805965488:85 , srv1827385525:85 , srv1833976775:85 , srv1881211681:85 , srv1963069380:85 , srv1966145979:85 , srv1989757419:85 , srv2095796285:85 , srv270445798:85 , srv368497152:85 , srv425323312:85 , srv439452672:85 , srv542025396:85 , srv580839121:85 , srv599451029:85 , srv608334095:85 , srv687192818:85 , srv831364257:85 , srv851478610:85 , srv863300430:85 , srv887869959:85 , srv892277310:85 , srv942932168:85 , srv948522465:85 , srv959054195:85 , srv978760109:85 , srv997386695:85 , srv444596947:86 , srv1446255634:639 } 2023-10-12 21:57:07,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 21:57:07,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2103058891=56, srv1599100141=27, srv1526861986=24, srv997386695=99, srv456907836=69, srv978760109=98, srv1090247101=4, srv1966145979=50, srv1890525517=44, srv1446255634=18, srv1451518071=19, srv523706886=71, srv771234423=82, srv332058768=61, srv1282846454=8, srv146861537=20, srv787363269=84, srv599451029=76, srv1394154233=16, srv959054195=97, srv382824755=63, srv1321823423=13, srv1432740081=17, srv2062869808=53, srv892277310=91, srv1352891095=14, srv1989757419=51, srv444596947=67, srv851478610=87, srv948522465=96, srv1931225420=48, srv1566462628=25, srv2095796285=55, srv323789886=60, srv439452672=66, srv580839121=75, srv771266679=83, srv425323312=65, srv856169262=88, srv26112956=57, srv608334095=77, srv1064965334=2, srv1619693807=28, srv1804224077=39, srv887869959=90, srv614113403=78, srv926692834=93, srv1721206675=34, srv1474413758=21, srv542025396=73, srv512992635=70, srv105924456=1, srv536202956=72, srv1311638490=12, srv1181079001=5, srv642572020=79, srv687192818=81, srv894016511=92, srv1726097686=35, srv938045976=94, srv2095120020=54, srv1827385525=41, srv108714086=3, srv139232624=15, srv1518250624=23, srv190775053=47, srv569247299=74, srv368497152=62, srv189258586=45, srv798283497=85, srv1833976775=42, srv1659880464=30, srv1805965488=40, srv29755771=59, srv1698761603=33, srv863300430=89, srv412818934=64, srv1003139334=0, srv1191015072=6, srv1794452657=38, srv1197979915=7, srv1781234293=36, srv148433082=22, srv942932168=95, srv166168422=31, srv1881211681=43, srv456440658=68, srv1622861453=29, srv1311606431=11, srv270445798=58, srv1683287280=32, srv1963069380=49, srv651691339=80, srv1899963456=46, srv1792503217=37, srv1990107384=52, srv1286163371=9, srv1592106032=26, srv1288552317=10, srv831364257=86} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 21:57:07,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 21:57:07,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 21:57:07,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 21:57:07,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 21:57:07,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 21:57:07,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 21:57:07,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 21:57:07,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 21:57:07,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 21:57:07,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 21:57:07,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 21:57:07,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 21:57:07,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 21:57:07,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 21:57:07,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 21:57:07,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 21:57:07,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 21:57:07,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 21:57:07,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 21:57:07,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 21:57:07,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 21:57:07,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 21:57:07,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 21:57:07,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 21:57:07,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 21:57:07,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 21:57:07,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 21:57:07,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 21:57:07,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 21:57:07,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 21:57:07,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-12 21:57:07,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-12 21:57:07,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-12 21:57:07,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-12 21:57:07,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-12 21:57:07,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-12 21:57:07,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-12 21:57:07,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-12 21:57:07,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-12 21:57:07,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-12 21:57:07,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-12 21:57:07,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-12 21:57:07,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-12 21:57:07,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-12 21:57:07,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-12 21:57:07,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-12 21:57:07,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-12 21:57:07,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-12 21:57:07,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-12 21:57:07,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-12 21:57:07,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-12 21:57:07,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-12 21:57:07,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-12 21:57:07,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-12 21:57:07,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-12 21:57:07,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-12 21:57:07,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-12 21:57:07,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-12 21:57:07,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-12 21:57:07,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-12 21:57:07,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-12 21:57:07,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-12 21:57:07,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-12 21:57:07,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-12 21:57:07,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-12 21:57:07,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-12 21:57:07,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-12 21:57:07,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-12 21:57:07,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-12 21:57:07,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-12 21:57:07,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-12 21:57:07,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-12 21:57:07,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-12 21:57:07,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-12 21:57:07,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-12 21:57:07,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-12 21:57:07,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-12 21:57:07,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-12 21:57:07,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-12 21:57:07,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-12 21:57:07,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-12 21:57:07,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-12 21:57:07,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-12 21:57:07,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-12 21:57:07,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-12 21:57:07,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-12 21:57:07,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-12 21:57:07,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-12 21:57:07,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-12 21:57:07,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-12 21:57:07,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-12 21:57:07,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-12 21:57:07,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-12 21:57:07,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-12 21:57:07,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-12 21:57:07,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-12 21:57:07,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-12 21:57:07,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-12 21:57:07,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-12 21:57:07,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-12 21:57:07,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-12 21:57:07,885 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.6207544128286207E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 21:57:07,885 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv146861537:83 , srv1781234293:83 , srv1931225420:83 , srv2062869808:83 , srv332058768:83 , srv798283497:83 , srv856169262:83 , srv108714086:84 , srv1090247101:84 , srv1181079001:84 , srv1282846454:84 , srv1286163371:84 , srv1311606431:84 , srv139232624:84 , srv1394154233:84 , srv1451518071:84 , srv1659880464:84 , srv166168422:84 , srv1698761603:84 , srv1721206675:84 , srv1804224077:84 , srv1890525517:84 , srv189258586:84 , srv1899963456:84 , srv190775053:84 , srv1990107384:84 , srv2095120020:84 , srv2103058891:84 , srv26112956:84 , srv29755771:84 , srv323789886:84 , srv382824755:84 , srv412818934:84 , srv456440658:84 , srv456907836:84 , srv512992635:84 , srv523706886:84 , srv536202956:84 , srv569247299:84 , srv614113403:84 , srv642572020:84 , srv651691339:84 , srv771234423:84 , srv771266679:84 , srv787363269:84 , srv894016511:84 , srv926692834:84 , srv938045976:84 , srv1003139334:85 , srv105924456:85 , srv1064965334:85 , srv1191015072:85 , srv1197979915:85 , srv1288552317:85 , srv1311638490:85 , srv1321823423:85 , srv1352891095:85 , srv1432740081:85 , srv1474413758:85 , srv148433082:85 , srv1518250624:85 , srv1526861986:85 , srv1566462628:85 , srv1592106032:85 , srv1599100141:85 , srv1619693807:85 , srv1622861453:85 , srv1683287280:85 , srv1726097686:85 , srv1792503217:85 , srv1794452657:85 , srv1805965488:85 , srv1827385525:85 , srv1833976775:85 , srv1881211681:85 , srv1963069380:85 , srv1966145979:85 , srv1989757419:85 , srv2095796285:85 , srv270445798:85 , srv368497152:85 , srv425323312:85 , srv439452672:85 , srv542025396:85 , srv580839121:85 , srv599451029:85 , srv608334095:85 , srv687192818:85 , srv831364257:85 , srv851478610:85 , srv863300430:85 , srv887869959:85 , srv892277310:85 , srv942932168:85 , srv948522465:85 , srv959054195:85 , srv978760109:85 , srv997386695:85 , srv444596947:86 , srv1446255634:639 } 2023-10-12 21:57:07,909 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=311 (was 311), ProcessCount=167 (was 167), AvailableMemoryMB=8698 (was 9243) 2023-10-12 21:57:07,915 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=311, ProcessCount=167, AvailableMemoryMB=8698 2023-10-12 21:57:07,916 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-12 21:57:07,916 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-12 21:57:07,923 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000361393:84 , srv1066244806:84 , srv1236533660:84 , srv1241373289:84 , srv1366427457:84 , srv1404126467:84 , srv151416162:84 , srv1610349449:84 , srv1665961239:84 , srv166676171:84 , srv1761087644:84 , srv1810655189:84 , srv1827222122:84 , srv1902963350:84 , srv1917663419:84 , srv1949198818:84 , srv1982527644:84 , srv2100313815:84 , srv2111217545:84 , srv304587208:84 , srv315422041:84 , srv387812427:84 , srv605430136:84 , srv60593284:84 , srv611826109:84 , srv653833244:84 , srv746058108:84 , srv758196414:84 , srv968042355:84 , srv851323996:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-12 21:57:07,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 21:57:07,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv315422041=20, srv758196414=27, srv746058108=26, srv1902963350=13, srv1810655189=11, srv2100313815=17, srv851323996=28, srv1241373289=3, srv304587208=19, srv166676171=9, srv1610349449=7, srv1665961239=8, srv60593284=23, srv1236533660=2, srv1982527644=16, srv151416162=6, srv1827222122=12, srv2111217545=18, srv387812427=21, srv1366427457=4, srv1761087644=10, srv1404126467=5, srv1000361393=0, srv605430136=22, srv653833244=25, srv1917663419=14, srv1949198818=15, srv968042355=29, srv1066244806=1, srv611826109=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 21:57:07,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 21:57:07,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 21:57:07,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 21:57:07,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 21:57:07,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 21:57:07,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 21:57:07,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 21:57:07,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 21:57:07,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 21:57:07,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 21:57:07,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-12 21:57:07,947 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1308469714943437, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 21:57:07,948 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912605337798911, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1308469714943437, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-12 21:59:07,936 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9869653 different iterations. Found a solution that moves 1836 regions; Going from a computed imbalance of 0.9912605337798911 to a new imbalance of 4.2853155918867096E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.68, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 21:59:07,942 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1000361393:90 , srv1066244806:90 , srv1236533660:90 , srv1241373289:90 , srv1366427457:90 , srv1404126467:90 , srv151416162:90 , srv1610349449:90 , srv1665961239:90 , srv166676171:90 , srv1761087644:90 , srv1810655189:90 , srv1827222122:90 , srv1902963350:90 , srv1917663419:90 , srv1949198818:90 , srv1982527644:90 , srv2100313815:90 , srv2111217545:90 , srv304587208:90 , srv315422041:90 , srv387812427:90 , srv605430136:90 , srv60593284:90 , srv611826109:90 , srv653833244:90 , srv746058108:90 , srv758196414:90 , srv851323996:90 , srv968042355:90 } 2023-10-12 21:59:07,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-12 21:59:07,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv315422041=20, srv758196414=27, srv746058108=26, srv1902963350=13, srv1810655189=11, srv2100313815=17, srv851323996=28, srv1241373289=3, srv304587208=19, srv166676171=9, srv1610349449=7, srv1665961239=8, srv60593284=23, srv1236533660=2, srv1982527644=16, srv151416162=6, srv1827222122=12, srv2111217545=18, srv387812427=21, srv1366427457=4, srv1761087644=10, srv1404126467=5, srv1000361393=0, srv605430136=22, srv653833244=25, srv1917663419=14, srv1949198818=15, srv968042355=29, srv1066244806=1, srv611826109=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-12 21:59:07,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-12 21:59:07,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-12 21:59:07,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-12 21:59:07,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-12 21:59:07,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-12 21:59:07,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-12 21:59:07,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-12 21:59:07,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-12 21:59:07,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-12 21:59:07,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-12 21:59:07,957 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-12 21:59:07,957 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1000361393:90 , srv1066244806:90 , srv1236533660:90 , srv1241373289:90 , srv1366427457:90 , srv1404126467:90 , srv151416162:90 , srv1610349449:90 , srv1665961239:90 , srv166676171:90 , srv1761087644:90 , srv1810655189:90 , srv1827222122:90 , srv1902963350:90 , srv1917663419:90 , srv1949198818:90 , srv1982527644:90 , srv2100313815:90 , srv2111217545:90 , srv304587208:90 , srv315422041:90 , srv387812427:90 , srv605430136:90 , srv60593284:90 , srv611826109:90 , srv653833244:90 , srv746058108:90 , srv758196414:90 , srv851323996:90 , srv968042355:90 } 2023-10-12 21:59:07,967 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=151 (was 311), ProcessCount=164 (was 167), AvailableMemoryMB=9454 (was 8698) - AvailableMemoryMB LEAK? -