2023-12-14 10:53:24,893 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-14 10:53:25,125 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-14 10:53:25,225 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-14 10:53:25,278 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-14 10:53:25,284 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-14 10:53:25,295 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=472, ProcessCount=173, AvailableMemoryMB=5373 2023-12-14 10:53:25,297 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-14 10:53:25,297 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-14 10:53:25,401 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1024210313:84 , srv1028427816:84 , srv1032770995:84 , srv1036160746:84 , srv1066746834:84 , srv107042578:84 , srv1073525543:84 , srv1074704445:84 , srv1079876823:84 , srv108182471:84 , srv1100435974:84 , srv1108435265:84 , srv1110126295:84 , srv1127557409:84 , srv1139734742:84 , srv1154850823:84 , srv1163128495:84 , srv1188719699:84 , srv1198944335:84 , srv1310299496:84 , srv1320680927:84 , srv1322114971:84 , srv1338616730:84 , srv1344750849:84 , srv1348621579:84 , srv1351429855:84 , srv1360649280:84 , srv1396455359:84 , srv1396911971:84 , srv1402334650:84 , srv1448721392:84 , srv1453372041:84 , srv1550091101:84 , srv1604299173:84 , srv1627850504:84 , srv1634446716:84 , srv1635297675:84 , srv1672372055:84 , srv1707202593:84 , srv1717890574:84 , srv1727262129:84 , srv1730234697:84 , srv1772299433:84 , srv1774022005:84 , srv1780823212:84 , srv1808207609:84 , srv1829779009:84 , srv1865877623:84 , srv1884618377:84 , srv1963020016:84 , srv196343798:84 , srv198021044:84 , srv2003178052:84 , srv2007944104:84 , srv2010779838:84 , srv2024046258:84 , srv2045111092:84 , srv2064826612:84 , srv2115223311:84 , srv2119338772:84 , srv2140732572:84 , srv21540723:84 , srv325286722:84 , srv325737984:84 , srv351231437:84 , srv357982156:84 , srv381986316:84 , srv405657833:84 , srv427062842:84 , srv447252320:84 , srv491388821:84 , srv49486615:84 , srv503778176:84 , srv506947115:84 , srv521230050:84 , srv525029387:84 , srv543271338:84 , srv54673333:84 , srv600845280:84 , srv604099467:84 , srv631352904:84 , srv672916097:84 , srv697594734:84 , srv698005704:84 , srv721840421:84 , srv723489235:84 , srv728231745:84 , srv736142253:84 , srv756555845:84 , srv76770951:84 , srv768056512:84 , srv779434235:84 , srv785695850:84 , srv790492335:84 , srv823771525:84 , srv85768974:84 , srv857691494:84 , srv861149365:84 , srv906491287:84 , srv498657281:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-14 10:53:25,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-14 10:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1154850823=15, srv198021044=51, srv1396911971=28, srv861149365=98, srv1829779009=46, srv728231745=87, srv631352904=81, srv506947115=74, srv1634446716=35, srv1727262129=40, srv1707202593=38, srv1024210313=0, srv1079876823=8, srv790492335=94, srv1338616730=22, srv1032770995=2, srv1344750849=23, srv1351429855=25, srv1066746834=4, srv1139734742=14, srv698005704=84, srv1774022005=43, srv2010779838=54, srv2119338772=59, srv427062842=68, srv76770951=90, srv498657281=72, srv1100435974=10, srv351231437=64, srv1963020016=49, srv2140732572=60, srv1127557409=13, srv1163128495=16, srv1028427816=1, srv1717890574=39, srv357982156=65, srv325286722=62, srv107042578=5, srv1320680927=20, srv2007944104=53, srv381986316=66, srv1772299433=42, srv1604299173=33, srv672916097=82, srv491388821=70, srv447252320=69, srv1310299496=19, srv2003178052=52, srv108182471=9, srv2045111092=56, srv521230050=75, srv723489235=86, srv2115223311=58, srv779434235=92, srv196343798=50, srv543271338=77, srv785695850=93, srv721840421=85, srv1073525543=6, srv823771525=95, srv600845280=79, srv1402334650=29, srv525029387=76, srv1198944335=18, srv697594734=83, srv1322114971=21, srv1348621579=24, srv2064826612=57, srv2024046258=55, srv1108435265=11, srv1036160746=3, srv1730234697=41, srv1808207609=45, srv49486615=71, srv1627850504=34, srv1074704445=7, srv325737984=63, srv405657833=67, srv1884618377=48, srv756555845=89, srv1448721392=30, srv604099467=80, srv1672372055=37, srv85768974=96, srv503778176=73, srv1550091101=32, srv857691494=97, srv1110126295=12, srv1453372041=31, srv21540723=61, srv54673333=78, srv736142253=88, srv1780823212=44, srv1635297675=36, srv906491287=99, srv1360649280=26, srv768056512=91, srv1396455359=27, srv1188719699=17, srv1865877623=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-14 10:53:25,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-14 10:53:25,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-14 10:53:25,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-14 10:53:25,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-14 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-14 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-14 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-14 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-14 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-14 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-14 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-14 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-14 10:53:25,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-14 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-14 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-14 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-14 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-14 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-14 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-14 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-14 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-14 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-14 10:53:25,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-14 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-14 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-14 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-14 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-14 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-14 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-14 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-14 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-14 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-14 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-14 10:53:25,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-14 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-14 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-14 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-14 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-14 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-14 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-14 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-14 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-14 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-14 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-14 10:53:25,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-14 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-14 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-14 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-14 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-14 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-14 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-14 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-14 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-14 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-14 10:53:25,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-14 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-14 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-14 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-14 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-14 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-14 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-14 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-14 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-14 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-14 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-14 10:53:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-14 10:53:25,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-14 10:53:25,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-14 10:53:25,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-14 10:53:25,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-14 10:53:25,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-14 10:53:25,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-14 10:53:25,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-14 10:53:25,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-14 10:53:25,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-14 10:53:25,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-14 10:53:25,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-14 10:53:25,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-14 10:53:25,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-14 10:53:25,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-14 10:53:25,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-14 10:53:25,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-14 10:53:25,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-14 10:53:25,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-14 10:53:25,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-14 10:53:25,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-14 10:53:25,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-14 10:53:25,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-14 10:53:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-14 10:53:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-14 10:53:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-14 10:53:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-14 10:53:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-14 10:53:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-14 10:53:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-14 10:53:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-14 10:53:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-14 10:53:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-14 10:53:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-14 10:53:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-14 10:53:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-14 10:53:25,626 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:53:25,626 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-14 10:53:25,627 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-14 10:53:26,092 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 581 ms to try 5000 different iterations. Found a solution that moves 4738 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17079817568785252. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04272312015332407, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0554139609729089, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5264444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04272312015332407, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.166, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23175, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:53:26,103 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1139734742:82 , srv1453372041:82 , srv2119338772:82 , srv1396455359:83 , srv1634446716:83 , srv1707202593:83 , srv1730234697:83 , srv2064826612:83 , srv672916097:83 , srv697594734:83 , srv721840421:83 , srv756555845:83 , srv76770951:83 , srv1073525543:84 , srv1079876823:84 , srv1100435974:84 , srv1310299496:84 , srv1627850504:84 , srv1865877623:84 , srv405657833:84 , srv600845280:84 , srv723489235:84 , srv728231745:84 , srv768056512:84 , srv906491287:84 , srv1036160746:85 , srv107042578:85 , srv1188719699:85 , srv1550091101:85 , srv1604299173:85 , srv1727262129:85 , srv1808207609:85 , srv198021044:85 , srv2007944104:85 , srv2045111092:85 , srv503778176:85 , srv506947115:85 , srv698005704:85 , srv779434235:85 , srv861149365:85 , srv1028427816:86 , srv1074704445:86 , srv1108435265:86 , srv1127557409:86 , srv1154850823:86 , srv1322114971:86 , srv1338616730:86 , srv1344750849:86 , srv1396911971:86 , srv1672372055:86 , srv1717890574:86 , srv1774022005:86 , srv21540723:86 , srv447252320:86 , srv491388821:86 , srv521230050:86 , srv54673333:86 , srv823771525:86 , srv1024210313:87 , srv1320680927:87 , srv1351429855:87 , srv1635297675:87 , srv1780823212:87 , srv196343798:87 , srv2115223311:87 , srv2140732572:87 , srv325737984:87 , srv604099467:87 , srv785695850:87 , srv1032770995:88 , srv1066746834:88 , srv108182471:88 , srv1110126295:88 , srv1348621579:88 , srv1360649280:88 , srv1448721392:88 , srv1772299433:88 , srv1884618377:88 , srv1963020016:88 , srv2010779838:88 , srv325286722:88 , srv351231437:88 , srv381986316:88 , srv49486615:88 , srv525029387:88 , srv543271338:88 , srv631352904:88 , srv1402334650:89 , srv2024046258:89 , srv357982156:89 , srv736142253:89 , srv790492335:89 , srv1163128495:90 , srv1198944335:90 , srv427062842:90 , srv1829779009:91 , srv857691494:91 , srv85768974:92 , srv2003178052:93 , srv498657281:470 } 2023-12-14 10:53:26,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-14 10:53:26,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1154850823=15, srv198021044=51, srv1396911971=28, srv861149365=98, srv1829779009=46, srv728231745=87, srv631352904=81, srv506947115=74, srv1634446716=35, srv1727262129=40, srv1707202593=38, srv1024210313=0, srv1079876823=8, srv790492335=94, srv1338616730=22, srv1032770995=2, srv1344750849=23, srv1351429855=25, srv1066746834=4, srv1139734742=14, srv698005704=84, srv1774022005=43, srv2010779838=54, srv2119338772=59, srv427062842=68, srv76770951=90, srv498657281=72, srv1100435974=10, srv351231437=64, srv1963020016=49, srv2140732572=60, srv1127557409=13, srv1163128495=16, srv1028427816=1, srv1717890574=39, srv357982156=65, srv325286722=62, srv107042578=5, srv1320680927=20, srv2007944104=53, srv381986316=66, srv1772299433=42, srv1604299173=33, srv672916097=82, srv491388821=70, srv447252320=69, srv1310299496=19, srv2003178052=52, srv108182471=9, srv2045111092=56, srv521230050=75, srv723489235=86, srv2115223311=58, srv779434235=92, srv196343798=50, srv543271338=77, srv785695850=93, srv721840421=85, srv1073525543=6, srv823771525=95, srv600845280=79, srv1402334650=29, srv525029387=76, srv1198944335=18, srv697594734=83, srv1322114971=21, srv1348621579=24, srv2064826612=57, srv2024046258=55, srv1108435265=11, srv1036160746=3, srv1730234697=41, srv1808207609=45, srv49486615=71, srv1627850504=34, srv1074704445=7, srv325737984=63, srv405657833=67, srv1884618377=48, srv756555845=89, srv1448721392=30, srv604099467=80, srv1672372055=37, srv85768974=96, srv503778176=73, srv1550091101=32, srv857691494=97, srv1110126295=12, srv1453372041=31, srv21540723=61, srv54673333=78, srv736142253=88, srv1780823212=44, srv1635297675=36, srv906491287=99, srv1360649280=26, srv768056512=91, srv1396455359=27, srv1188719699=17, srv1865877623=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-14 10:53:26,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-14 10:53:26,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-14 10:53:26,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-14 10:53:26,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-14 10:53:26,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-14 10:53:26,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-14 10:53:26,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-14 10:53:26,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-14 10:53:26,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-14 10:53:26,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-14 10:53:26,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-14 10:53:26,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-14 10:53:26,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-14 10:53:26,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-14 10:53:26,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-14 10:53:26,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-14 10:53:26,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-14 10:53:26,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-14 10:53:26,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-14 10:53:26,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-14 10:53:26,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-14 10:53:26,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-14 10:53:26,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-14 10:53:26,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-14 10:53:26,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-14 10:53:26,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-14 10:53:26,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-14 10:53:26,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-14 10:53:26,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-14 10:53:26,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-14 10:53:26,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-14 10:53:26,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-14 10:53:26,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-14 10:53:26,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-14 10:53:26,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-14 10:53:26,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-14 10:53:26,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-14 10:53:26,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-14 10:53:26,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-14 10:53:26,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-14 10:53:26,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-14 10:53:26,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-14 10:53:26,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-14 10:53:26,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-14 10:53:26,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-14 10:53:26,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-14 10:53:26,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-14 10:53:26,236 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04272312015332407, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0554139609729089, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04272312015332407, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.166, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23175, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:53:26,236 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-14 10:53:26,237 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1707649995027637, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04272312015332407, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0554139609729089, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04272312015332407, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.166, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23175, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-14 10:53:26,406 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 258 ms to try 5000 different iterations. Found a solution that moves 2593 regions; Going from a computed imbalance of 0.1707649995027637 to a new imbalance of 0.014842205958202386. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023396579347795755); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03485419364746249, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2881111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023396579347795755); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.041666666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:53:26,418 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv85768974:79 , srv1829779009:80 , srv857691494:81 , srv108182471:82 , srv1163128495:82 , srv2010779838:82 , srv357982156:82 , srv381986316:82 , srv427062842:82 , srv525029387:82 , srv543271338:82 , srv790492335:82 , srv1032770995:83 , srv1320680927:83 , srv1448721392:83 , srv1635297675:83 , srv196343798:83 , srv49486615:83 , srv1024210313:84 , srv1066746834:84 , srv1154850823:84 , srv1198944335:84 , srv1360649280:84 , srv1402334650:84 , srv1884618377:84 , srv1963020016:84 , srv2003178052:84 , srv325286722:84 , srv736142253:84 , srv785695850:84 , srv1127557409:85 , srv1344750849:85 , srv1348621579:85 , srv1351429855:85 , srv1550091101:85 , srv1772299433:85 , srv2115223311:85 , srv21540723:85 , srv325737984:85 , srv351231437:85 , srv54673333:85 , srv604099467:85 , srv631352904:85 , srv823771525:85 , srv861149365:85 , srv107042578:86 , srv1074704445:86 , srv1322114971:86 , srv1338616730:86 , srv1396911971:86 , srv1604299173:86 , srv1727262129:86 , srv1780823212:86 , srv2024046258:86 , srv2045111092:86 , srv2140732572:86 , srv447252320:86 , srv491388821:86 , srv506947115:86 , srv521230050:86 , srv768056512:86 , srv1073525543:87 , srv1100435974:87 , srv1108435265:87 , srv1110126295:87 , srv1627850504:87 , srv1672372055:87 , srv1774022005:87 , srv503778176:87 , srv698005704:87 , srv721840421:87 , srv723489235:87 , srv728231745:87 , srv76770951:87 , srv1028427816:88 , srv1036160746:88 , srv1079876823:88 , srv1139734742:88 , srv1188719699:88 , srv1453372041:88 , srv1707202593:88 , srv1717890574:88 , srv405657833:88 , srv697594734:88 , srv779434235:88 , srv906491287:88 , srv1396455359:89 , srv1808207609:89 , srv1865877623:89 , srv198021044:89 , srv2007944104:89 , srv2119338772:89 , srv600845280:89 , srv756555845:89 , srv1310299496:90 , srv2064826612:90 , srv1634446716:91 , srv1730234697:91 , srv672916097:92 , srv498657281:512 } 2023-12-14 10:53:26,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-14 10:53:26,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1154850823=15, srv198021044=51, srv1396911971=28, srv861149365=98, srv1829779009=46, srv728231745=87, srv631352904=81, srv506947115=74, srv1634446716=35, srv1727262129=40, srv1707202593=38, srv1024210313=0, srv1079876823=8, srv790492335=94, srv1338616730=22, srv1032770995=2, srv1344750849=23, srv1351429855=25, srv1066746834=4, srv1139734742=14, srv698005704=84, srv1774022005=43, srv2010779838=54, srv2119338772=59, srv427062842=68, srv76770951=90, srv498657281=72, srv1100435974=10, srv351231437=64, srv1963020016=49, srv2140732572=60, srv1127557409=13, srv1163128495=16, srv1028427816=1, srv1717890574=39, srv357982156=65, srv325286722=62, srv107042578=5, srv1320680927=20, srv2007944104=53, srv381986316=66, srv1772299433=42, srv1604299173=33, srv672916097=82, srv491388821=70, srv447252320=69, srv1310299496=19, srv2003178052=52, srv108182471=9, srv2045111092=56, srv521230050=75, srv723489235=86, srv2115223311=58, srv779434235=92, srv196343798=50, srv543271338=77, srv785695850=93, srv721840421=85, srv1073525543=6, srv823771525=95, srv600845280=79, srv1402334650=29, srv525029387=76, srv1198944335=18, srv697594734=83, srv1322114971=21, srv1348621579=24, srv2064826612=57, srv2024046258=55, srv1108435265=11, srv1036160746=3, srv1730234697=41, srv1808207609=45, srv49486615=71, srv1627850504=34, srv1074704445=7, srv325737984=63, srv405657833=67, srv1884618377=48, srv756555845=89, srv1448721392=30, srv604099467=80, srv1672372055=37, srv85768974=96, srv503778176=73, srv1550091101=32, srv857691494=97, srv1110126295=12, srv1453372041=31, srv21540723=61, srv54673333=78, srv736142253=88, srv1780823212=44, srv1635297675=36, srv906491287=99, srv1360649280=26, srv768056512=91, srv1396455359=27, srv1188719699=17, srv1865877623=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-14 10:53:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-14 10:53:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-14 10:53:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-14 10:53:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-14 10:53:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-14 10:53:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-14 10:53:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-14 10:53:26,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-14 10:53:26,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-14 10:53:26,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-14 10:53:26,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-14 10:53:26,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-14 10:53:26,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-14 10:53:26,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-14 10:53:26,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-14 10:53:26,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-14 10:53:26,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-14 10:53:26,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-14 10:53:26,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-14 10:53:26,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-14 10:53:26,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-14 10:53:26,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-14 10:53:26,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-14 10:53:26,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-14 10:53:26,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-14 10:53:26,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-14 10:53:26,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-14 10:53:26,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-14 10:53:26,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-14 10:53:26,536 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023396579347795755); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03485419364746249, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023396579347795755); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.041666666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:53:26,536 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-14 10:53:26,536 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014824049384134147, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023396579347795755); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03485419364746249, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023396579347795755); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.041666666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-14 10:53:26,672 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 210 ms to try 5000 different iterations. Found a solution that moves 968 regions; Going from a computed imbalance of 0.014824049384134147 to a new imbalance of 8.618262844204705E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006217570812329759); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00663291117550788); MoveCostFunction : (multiplier=7.0, imbalance=0.10755555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006217570812329759); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:53:26,674 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2003178052:80 , srv1110126295:83 , srv1188719699:83 , srv1198944335:83 , srv1717890574:83 , srv1730234697:83 , srv1808207609:83 , srv2007944104:83 , srv2024046258:83 , srv2064826612:83 , srv672916097:83 , srv1028427816:84 , srv1036160746:84 , srv1310299496:84 , srv1322114971:84 , srv1396455359:84 , srv1634446716:84 , srv1774022005:84 , srv198021044:84 , srv2140732572:84 , srv351231437:84 , srv600845280:84 , srv1074704445:85 , srv1079876823:85 , srv1163128495:85 , srv1338616730:85 , srv1348621579:85 , srv1396911971:85 , srv1402334650:85 , srv1672372055:85 , srv1772299433:85 , srv1963020016:85 , srv325286722:85 , srv405657833:85 , srv427062842:85 , srv521230050:85 , srv604099467:85 , srv631352904:85 , srv698005704:85 , srv736142253:85 , srv756555845:85 , srv779434235:85 , srv857691494:85 , srv1024210313:86 , srv1032770995:86 , srv1066746834:86 , srv107042578:86 , srv1100435974:86 , srv1108435265:86 , srv1127557409:86 , srv1351429855:86 , srv1360649280:86 , srv1448721392:86 , srv1604299173:86 , srv1627850504:86 , srv1780823212:86 , srv1829779009:86 , srv1865877623:86 , srv1884618377:86 , srv196343798:86 , srv2045111092:86 , srv2115223311:86 , srv21540723:86 , srv325737984:86 , srv447252320:86 , srv491388821:86 , srv503778176:86 , srv506947115:86 , srv723489235:86 , srv728231745:86 , srv785695850:86 , srv85768974:86 , srv906491287:86 , srv1073525543:87 , srv108182471:87 , srv1154850823:87 , srv1320680927:87 , srv1344750849:87 , srv1635297675:87 , srv1727262129:87 , srv2119338772:87 , srv357982156:87 , srv381986316:87 , srv49486615:87 , srv543271338:87 , srv54673333:87 , srv697594734:87 , srv721840421:87 , srv76770951:87 , srv790492335:87 , srv823771525:87 , srv861149365:87 , srv1139734742:88 , srv1453372041:88 , srv1707202593:88 , srv2010779838:88 , srv525029387:88 , srv768056512:88 , srv1550091101:89 , srv498657281:531 } 2023-12-14 10:53:26,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-14 10:53:26,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1154850823=15, srv198021044=51, srv1396911971=28, srv861149365=98, srv1829779009=46, srv728231745=87, srv631352904=81, srv506947115=74, srv1634446716=35, srv1727262129=40, srv1707202593=38, srv1024210313=0, srv1079876823=8, srv790492335=94, srv1338616730=22, srv1032770995=2, srv1344750849=23, srv1351429855=25, srv1066746834=4, srv1139734742=14, srv698005704=84, srv1774022005=43, srv2010779838=54, srv2119338772=59, srv427062842=68, srv76770951=90, srv498657281=72, srv1100435974=10, srv351231437=64, srv1963020016=49, srv2140732572=60, srv1127557409=13, srv1163128495=16, srv1028427816=1, srv1717890574=39, srv357982156=65, srv325286722=62, srv107042578=5, srv1320680927=20, srv2007944104=53, srv381986316=66, srv1772299433=42, srv1604299173=33, srv672916097=82, srv491388821=70, srv447252320=69, srv1310299496=19, srv2003178052=52, srv108182471=9, srv2045111092=56, srv521230050=75, srv723489235=86, srv2115223311=58, srv779434235=92, srv196343798=50, srv543271338=77, srv785695850=93, srv721840421=85, srv1073525543=6, srv823771525=95, srv600845280=79, srv1402334650=29, srv525029387=76, srv1198944335=18, srv697594734=83, srv1322114971=21, srv1348621579=24, srv2064826612=57, srv2024046258=55, srv1108435265=11, srv1036160746=3, srv1730234697=41, srv1808207609=45, srv49486615=71, srv1627850504=34, srv1074704445=7, srv325737984=63, srv405657833=67, srv1884618377=48, srv756555845=89, srv1448721392=30, srv604099467=80, srv1672372055=37, srv85768974=96, srv503778176=73, srv1550091101=32, srv857691494=97, srv1110126295=12, srv1453372041=31, srv21540723=61, srv54673333=78, srv736142253=88, srv1780823212=44, srv1635297675=36, srv906491287=99, srv1360649280=26, srv768056512=91, srv1396455359=27, srv1188719699=17, srv1865877623=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-14 10:53:26,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-14 10:53:26,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-14 10:53:26,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-14 10:53:26,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-14 10:53:26,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-14 10:53:26,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-14 10:53:26,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-14 10:53:26,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-14 10:53:26,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-14 10:53:26,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-14 10:53:26,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-14 10:53:26,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-14 10:53:26,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-14 10:53:26,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-14 10:53:26,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-14 10:53:26,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-14 10:53:26,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-14 10:53:26,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-14 10:53:26,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-14 10:53:26,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-14 10:53:26,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-14 10:53:26,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-14 10:53:26,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-14 10:53:26,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-14 10:53:26,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-14 10:53:26,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-14 10:53:26,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-14 10:53:26,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-14 10:53:26,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-14 10:53:26,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-14 10:53:26,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-14 10:53:26,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-14 10:53:26,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-14 10:53:26,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-14 10:53:26,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-14 10:53:26,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-14 10:53:26,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-14 10:53:26,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-14 10:53:26,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-14 10:53:26,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-14 10:53:26,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-14 10:53:26,772 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006217570812329759); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00663291117550788); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006217570812329759); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:53:26,772 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-14 10:53:26,772 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.55048203549643E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006217570812329759); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00663291117550788); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006217570812329759); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-14 10:53:26,978 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 277 ms to try 5000 different iterations. Found a solution that moves 266 regions; Going from a computed imbalance of 8.55048203549643E-4 to a new imbalance of 3.4500612965009627E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.029555555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:53:26,979 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1550091101:82 , srv1108435265:83 , srv1139734742:83 , srv1707202593:83 , srv1727262129:83 , srv1780823212:83 , srv1865877623:83 , srv823771525:83 , srv1024210313:84 , srv1028427816:84 , srv1066746834:84 , srv1073525543:84 , srv1110126295:84 , srv1310299496:84 , srv1344750849:84 , srv1348621579:84 , srv1351429855:84 , srv1360649280:84 , srv1396911971:84 , srv1453372041:84 , srv1634446716:84 , srv1635297675:84 , srv1672372055:84 , srv1772299433:84 , srv1884618377:84 , srv198021044:84 , srv2010779838:84 , srv2024046258:84 , srv2115223311:84 , srv2119338772:84 , srv357982156:84 , srv447252320:84 , srv491388821:84 , srv49486615:84 , srv503778176:84 , srv525029387:84 , srv54673333:84 , srv631352904:84 , srv697594734:84 , srv698005704:84 , srv736142253:84 , srv790492335:84 , srv906491287:84 , srv1032770995:85 , srv1036160746:85 , srv107042578:85 , srv1074704445:85 , srv1079876823:85 , srv108182471:85 , srv1100435974:85 , srv1127557409:85 , srv1154850823:85 , srv1163128495:85 , srv1198944335:85 , srv1320680927:85 , srv1402334650:85 , srv1448721392:85 , srv1604299173:85 , srv1627850504:85 , srv1717890574:85 , srv1730234697:85 , srv1774022005:85 , srv1829779009:85 , srv1963020016:85 , srv2003178052:85 , srv2007944104:85 , srv2045111092:85 , srv2140732572:85 , srv21540723:85 , srv325286722:85 , srv325737984:85 , srv351231437:85 , srv381986316:85 , srv405657833:85 , srv427062842:85 , srv506947115:85 , srv521230050:85 , srv543271338:85 , srv600845280:85 , srv604099467:85 , srv672916097:85 , srv723489235:85 , srv728231745:85 , srv756555845:85 , srv76770951:85 , srv768056512:85 , srv779434235:85 , srv785695850:85 , srv85768974:85 , srv857691494:85 , srv861149365:85 , srv1188719699:86 , srv1322114971:86 , srv1338616730:86 , srv1396455359:86 , srv1808207609:86 , srv196343798:86 , srv2064826612:86 , srv721840421:86 , srv498657281:629 } 2023-12-14 10:53:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-14 10:53:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1154850823=15, srv198021044=51, srv1396911971=28, srv861149365=98, srv1829779009=46, srv728231745=87, srv631352904=81, srv506947115=74, srv1634446716=35, srv1727262129=40, srv1707202593=38, srv1024210313=0, srv1079876823=8, srv790492335=94, srv1338616730=22, srv1032770995=2, srv1344750849=23, srv1351429855=25, srv1066746834=4, srv1139734742=14, srv698005704=84, srv1774022005=43, srv2010779838=54, srv2119338772=59, srv427062842=68, srv76770951=90, srv498657281=72, srv1100435974=10, srv351231437=64, srv1963020016=49, srv2140732572=60, srv1127557409=13, srv1163128495=16, srv1028427816=1, srv1717890574=39, srv357982156=65, srv325286722=62, srv107042578=5, srv1320680927=20, srv2007944104=53, srv381986316=66, srv1772299433=42, srv1604299173=33, srv672916097=82, srv491388821=70, srv447252320=69, srv1310299496=19, srv2003178052=52, srv108182471=9, srv2045111092=56, srv521230050=75, srv723489235=86, srv2115223311=58, srv779434235=92, srv196343798=50, srv543271338=77, srv785695850=93, srv721840421=85, srv1073525543=6, srv823771525=95, srv600845280=79, srv1402334650=29, srv525029387=76, srv1198944335=18, srv697594734=83, srv1322114971=21, srv1348621579=24, srv2064826612=57, srv2024046258=55, srv1108435265=11, srv1036160746=3, srv1730234697=41, srv1808207609=45, srv49486615=71, srv1627850504=34, srv1074704445=7, srv325737984=63, srv405657833=67, srv1884618377=48, srv756555845=89, srv1448721392=30, srv604099467=80, srv1672372055=37, srv85768974=96, srv503778176=73, srv1550091101=32, srv857691494=97, srv1110126295=12, srv1453372041=31, srv21540723=61, srv54673333=78, srv736142253=88, srv1780823212=44, srv1635297675=36, srv906491287=99, srv1360649280=26, srv768056512=91, srv1396455359=27, srv1188719699=17, srv1865877623=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-14 10:53:26,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-14 10:53:26,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-14 10:53:26,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-14 10:53:26,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-14 10:53:26,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-14 10:53:26,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-14 10:53:26,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-14 10:53:26,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-14 10:53:26,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-14 10:53:26,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-14 10:53:26,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-14 10:53:26,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-14 10:53:26,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-14 10:53:26,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-14 10:53:26,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-14 10:53:26,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-14 10:53:26,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-14 10:53:26,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-14 10:53:26,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-14 10:53:26,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-14 10:53:26,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-14 10:53:26,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-14 10:53:27,078 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:53:27,079 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-14 10:53:27,079 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.2638041155464095E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-14 10:53:27,213 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 210 ms to try 5000 different iterations. Found a solution that moves 28 regions; Going from a computed imbalance of 3.2638041155464095E-5 to a new imbalance of 6.2281389302068456E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.003111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:53:27,213 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1338616730:83 , srv1402334650:83 , srv1808207609:83 , srv325737984:83 , srv672916097:83 , srv721840421:83 , srv768056512:83 , srv779434235:83 , srv1028427816:84 , srv1032770995:84 , srv1036160746:84 , srv1066746834:84 , srv107042578:84 , srv1073525543:84 , srv1074704445:84 , srv1079876823:84 , srv108182471:84 , srv1100435974:84 , srv1108435265:84 , srv1110126295:84 , srv1127557409:84 , srv1154850823:84 , srv1163128495:84 , srv1188719699:84 , srv1198944335:84 , srv1310299496:84 , srv1320680927:84 , srv1322114971:84 , srv1344750849:84 , srv1348621579:84 , srv1351429855:84 , srv1360649280:84 , srv1396455359:84 , srv1448721392:84 , srv1453372041:84 , srv1604299173:84 , srv1627850504:84 , srv1634446716:84 , srv1672372055:84 , srv1707202593:84 , srv1717890574:84 , srv1730234697:84 , srv1772299433:84 , srv1774022005:84 , srv1780823212:84 , srv1829779009:84 , srv1865877623:84 , srv1884618377:84 , srv1963020016:84 , srv196343798:84 , srv198021044:84 , srv2003178052:84 , srv2007944104:84 , srv2010779838:84 , srv2024046258:84 , srv2045111092:84 , srv2064826612:84 , srv2115223311:84 , srv2119338772:84 , srv2140732572:84 , srv21540723:84 , srv325286722:84 , srv351231437:84 , srv357982156:84 , srv381986316:84 , srv405657833:84 , srv427062842:84 , srv447252320:84 , srv491388821:84 , srv49486615:84 , srv503778176:84 , srv506947115:84 , srv521230050:84 , srv525029387:84 , srv543271338:84 , srv54673333:84 , srv600845280:84 , srv604099467:84 , srv631352904:84 , srv697594734:84 , srv723489235:84 , srv728231745:84 , srv736142253:84 , srv756555845:84 , srv76770951:84 , srv785695850:84 , srv790492335:84 , srv85768974:84 , srv857691494:84 , srv861149365:84 , srv906491287:84 , srv1024210313:85 , srv1139734742:85 , srv1396911971:85 , srv1550091101:85 , srv1635297675:85 , srv1727262129:85 , srv698005704:85 , srv823771525:85 , srv498657281:684 } 2023-12-14 10:53:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-14 10:53:27,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1154850823=15, srv198021044=51, srv1396911971=28, srv861149365=98, srv1829779009=46, srv728231745=87, srv631352904=81, srv506947115=74, srv1634446716=35, srv1727262129=40, srv1707202593=38, srv1024210313=0, srv1079876823=8, srv790492335=94, srv1338616730=22, srv1032770995=2, srv1344750849=23, srv1351429855=25, srv1066746834=4, srv1139734742=14, srv698005704=84, srv1774022005=43, srv2010779838=54, srv2119338772=59, srv427062842=68, srv76770951=90, srv498657281=72, srv1100435974=10, srv351231437=64, srv1963020016=49, srv2140732572=60, srv1127557409=13, srv1163128495=16, srv1028427816=1, srv1717890574=39, srv357982156=65, srv325286722=62, srv107042578=5, srv1320680927=20, srv2007944104=53, srv381986316=66, srv1772299433=42, srv1604299173=33, srv672916097=82, srv491388821=70, srv447252320=69, srv1310299496=19, srv2003178052=52, srv108182471=9, srv2045111092=56, srv521230050=75, srv723489235=86, srv2115223311=58, srv779434235=92, srv196343798=50, srv543271338=77, srv785695850=93, srv721840421=85, srv1073525543=6, srv823771525=95, srv600845280=79, srv1402334650=29, srv525029387=76, srv1198944335=18, srv697594734=83, srv1322114971=21, srv1348621579=24, srv2064826612=57, srv2024046258=55, srv1108435265=11, srv1036160746=3, srv1730234697=41, srv1808207609=45, srv49486615=71, srv1627850504=34, srv1074704445=7, srv325737984=63, srv405657833=67, srv1884618377=48, srv756555845=89, srv1448721392=30, srv604099467=80, srv1672372055=37, srv85768974=96, srv503778176=73, srv1550091101=32, srv857691494=97, srv1110126295=12, srv1453372041=31, srv21540723=61, srv54673333=78, srv736142253=88, srv1780823212=44, srv1635297675=36, srv906491287=99, srv1360649280=26, srv768056512=91, srv1396455359=27, srv1188719699=17, srv1865877623=47} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-14 10:53:27,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-14 10:53:27,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-14 10:53:27,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-14 10:53:27,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-14 10:53:27,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-14 10:53:27,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-14 10:53:27,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-14 10:53:27,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-14 10:53:27,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-14 10:53:27,315 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.032078739728369E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:53:27,315 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1338616730:83 , srv1402334650:83 , srv1808207609:83 , srv325737984:83 , srv672916097:83 , srv721840421:83 , srv768056512:83 , srv779434235:83 , srv1028427816:84 , srv1032770995:84 , srv1036160746:84 , srv1066746834:84 , srv107042578:84 , srv1073525543:84 , srv1074704445:84 , srv1079876823:84 , srv108182471:84 , srv1100435974:84 , srv1108435265:84 , srv1110126295:84 , srv1127557409:84 , srv1154850823:84 , srv1163128495:84 , srv1188719699:84 , srv1198944335:84 , srv1310299496:84 , srv1320680927:84 , srv1322114971:84 , srv1344750849:84 , srv1348621579:84 , srv1351429855:84 , srv1360649280:84 , srv1396455359:84 , srv1448721392:84 , srv1453372041:84 , srv1604299173:84 , srv1627850504:84 , srv1634446716:84 , srv1672372055:84 , srv1707202593:84 , srv1717890574:84 , srv1730234697:84 , srv1772299433:84 , srv1774022005:84 , srv1780823212:84 , srv1829779009:84 , srv1865877623:84 , srv1884618377:84 , srv1963020016:84 , srv196343798:84 , srv198021044:84 , srv2003178052:84 , srv2007944104:84 , srv2010779838:84 , srv2024046258:84 , srv2045111092:84 , srv2064826612:84 , srv2115223311:84 , srv2119338772:84 , srv2140732572:84 , srv21540723:84 , srv325286722:84 , srv351231437:84 , srv357982156:84 , srv381986316:84 , srv405657833:84 , srv427062842:84 , srv447252320:84 , srv491388821:84 , srv49486615:84 , srv503778176:84 , srv506947115:84 , srv521230050:84 , srv525029387:84 , srv543271338:84 , srv54673333:84 , srv600845280:84 , srv604099467:84 , srv631352904:84 , srv697594734:84 , srv723489235:84 , srv728231745:84 , srv736142253:84 , srv756555845:84 , srv76770951:84 , srv785695850:84 , srv790492335:84 , srv85768974:84 , srv857691494:84 , srv861149365:84 , srv906491287:84 , srv1024210313:85 , srv1139734742:85 , srv1396911971:85 , srv1550091101:85 , srv1635297675:85 , srv1727262129:85 , srv698005704:85 , srv823771525:85 , srv498657281:684 } 2023-12-14 10:53:27,337 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=482 (was 472) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 173), AvailableMemoryMB=4879 (was 5373) 2023-12-14 10:53:27,344 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=482, ProcessCount=173, AvailableMemoryMB=4879 2023-12-14 10:53:27,344 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-14 10:53:27,344 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-14 10:53:27,352 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013610474:84 , srv1042965479:84 , srv1179643762:84 , srv1186469240:84 , srv1229538256:84 , srv1233308426:84 , srv125277349:84 , srv1253354526:84 , srv1329500907:84 , srv1380047894:84 , srv1385215601:84 , srv144201959:84 , srv1481697661:84 , srv161125407:84 , srv1626407812:84 , srv1672972043:84 , srv1695415604:84 , srv1936759994:84 , srv2100510383:84 , srv30002902:84 , srv617189621:84 , srv624614315:84 , srv640608975:84 , srv705876577:84 , srv738350327:84 , srv95477432:84 , srv980305406:84 , srv99262818:84 , srv995119642:84 , srv540143050:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-14 10:53:27,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-14 10:53:27,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv144201959=11, srv617189621=21, srv705876577=24, srv1186469240=3, srv1672972043=15, srv1253354526=7, srv995119642=29, srv99262818=28, srv1380047894=9, srv640608975=23, srv540143050=20, srv1233308426=5, srv738350327=25, srv1179643762=2, srv2100510383=18, srv1042965479=1, srv1936759994=17, srv624614315=22, srv1385215601=10, srv1695415604=16, srv980305406=27, srv125277349=6, srv1329500907=8, srv95477432=26, srv30002902=19, srv1626407812=14, srv161125407=13, srv1229538256=4, srv1481697661=12, srv1013610474=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-14 10:53:27,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-14 10:53:27,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-14 10:53:27,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-14 10:53:27,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-14 10:53:27,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-14 10:53:27,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-14 10:53:27,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-14 10:53:27,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-14 10:53:27,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-14 10:53:27,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-14 10:53:27,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-14 10:53:27,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-14 10:53:27,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-14 10:53:27,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-14 10:53:27,378 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1358357192956268, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:53:27,378 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912621057180336, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1358357192956268, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-14 10:55:27,366 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 11049580 different iterations. Found a solution that moves 1841 regions; Going from a computed imbalance of 0.9912621057180336 to a new imbalance of 4.296985841319951E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6818518518518518, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:55:27,373 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1013610474:90 , srv1042965479:90 , srv1179643762:90 , srv1186469240:90 , srv1229538256:90 , srv1233308426:90 , srv125277349:90 , srv1253354526:90 , srv1329500907:90 , srv1380047894:90 , srv1385215601:90 , srv144201959:90 , srv1481697661:90 , srv161125407:90 , srv1626407812:90 , srv1672972043:90 , srv1695415604:90 , srv1936759994:90 , srv2100510383:90 , srv30002902:90 , srv540143050:90 , srv617189621:90 , srv624614315:90 , srv640608975:90 , srv705876577:90 , srv738350327:90 , srv95477432:90 , srv980305406:90 , srv99262818:90 , srv995119642:90 } 2023-12-14 10:55:27,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-14 10:55:27,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv144201959=11, srv617189621=21, srv705876577=24, srv1186469240=3, srv1672972043=15, srv1253354526=7, srv995119642=29, srv99262818=28, srv1380047894=9, srv640608975=23, srv540143050=20, srv1233308426=5, srv738350327=25, srv1179643762=2, srv2100510383=18, srv1042965479=1, srv1936759994=17, srv624614315=22, srv1385215601=10, srv1695415604=16, srv980305406=27, srv125277349=6, srv1329500907=8, srv95477432=26, srv30002902=19, srv1626407812=14, srv161125407=13, srv1229538256=4, srv1481697661=12, srv1013610474=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-14 10:55:27,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-14 10:55:27,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-14 10:55:27,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-14 10:55:27,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-14 10:55:27,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-14 10:55:27,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-14 10:55:27,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-14 10:55:27,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-14 10:55:27,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-14 10:55:27,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-14 10:55:27,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-14 10:55:27,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-14 10:55:27,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-14 10:55:27,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-14 10:55:27,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-14 10:55:27,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-14 10:55:27,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-14 10:55:27,394 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-14 10:55:27,394 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1013610474:90 , srv1042965479:90 , srv1179643762:90 , srv1186469240:90 , srv1229538256:90 , srv1233308426:90 , srv125277349:90 , srv1253354526:90 , srv1329500907:90 , srv1380047894:90 , srv1385215601:90 , srv144201959:90 , srv1481697661:90 , srv161125407:90 , srv1626407812:90 , srv1672972043:90 , srv1695415604:90 , srv1936759994:90 , srv2100510383:90 , srv30002902:90 , srv540143050:90 , srv617189621:90 , srv624614315:90 , srv640608975:90 , srv705876577:90 , srv738350327:90 , srv95477432:90 , srv980305406:90 , srv99262818:90 , srv995119642:90 } 2023-12-14 10:55:27,411 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=215 (was 482), ProcessCount=171 (was 173), AvailableMemoryMB=4789 (was 4879)