2023-04-10 16:56:43,355 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-04-10 16:56:43,687 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-10 16:56:43,790 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-10 16:56:43,846 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-10 16:56:43,853 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-10 16:56:43,865 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=416, ProcessCount=169, AvailableMemoryMB=1370 2023-04-10 16:56:43,867 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-10 16:56:43,868 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-10 16:56:43,978 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100091796:84 , srv1018654109:84 , srv1025578333:84 , srv1029196969:84 , srv1061117445:84 , srv1071625643:84 , srv1073262092:84 , srv1101473436:84 , srv1143000408:84 , srv1173328186:84 , srv1203049086:84 , srv122097009:84 , srv1296444058:84 , srv1299933155:84 , srv1322495483:84 , srv1322682902:84 , srv135222558:84 , srv1359208880:84 , srv1427897215:84 , srv1455442750:84 , srv1476029536:84 , srv1489889413:84 , srv1539272226:84 , srv1539452449:84 , srv1565174562:84 , srv1601458961:84 , srv1612038212:84 , srv164106960:84 , srv1658741159:84 , srv1665431057:84 , srv1671189252:84 , srv1688659034:84 , srv1691073220:84 , srv1693546116:84 , srv1711671185:84 , srv1717976470:84 , srv1735830053:84 , srv1756084493:84 , srv1763408484:84 , srv1772683108:84 , srv1780323301:84 , srv1818291625:84 , srv181838762:84 , srv1833486111:84 , srv1841541305:84 , srv1863571168:84 , srv1866614417:84 , srv1884575519:84 , srv1887575737:84 , srv1903296531:84 , srv1914494785:84 , srv194297072:84 , srv195160449:84 , srv1965815862:84 , srv1966521433:84 , srv1990635220:84 , srv1993796595:84 , srv2031706853:84 , srv2032172788:84 , srv2045136361:84 , srv2110137983:84 , srv25323499:84 , srv291927189:84 , srv306421395:84 , srv30976376:84 , srv33480024:84 , srv338783135:84 , srv393745905:84 , srv421398282:84 , srv437342945:84 , srv441834510:84 , srv443868613:84 , srv452570705:84 , srv49323549:84 , srv502404229:84 , srv535732061:84 , srv538219435:84 , srv567656518:84 , srv581998937:84 , srv594057986:84 , srv617367867:84 , srv626739711:84 , srv6380927:84 , srv646786044:84 , srv648236756:84 , srv665030735:84 , srv673568557:84 , srv675048355:84 , srv68734350:84 , srv801602011:84 , srv819857485:84 , srv82998332:84 , srv832875297:84 , srv851879123:84 , srv85694144:84 , srv903895227:84 , srv95658054:84 , srv962765548:84 , srv981805548:84 , srv140694039:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-04-10 16:56:43,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-10 16:56:43,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv338783135=67, srv1990635220=56, srv291927189=63, srv819857485=91, srv1427897215=19, srv1903296531=50, srv1029196969=3, srv1612038212=27, srv95658054=97, srv441834510=71, srv1717976470=36, srv1018654109=1, srv135222558=16, srv665030735=86, srv981805548=99, srv1061117445=4, srv1763408484=39, srv1993796595=57, srv502404229=75, srv122097009=11, srv1359208880=17, srv1884575519=48, srv538219435=77, srv194297072=52, srv421398282=69, srv1735830053=37, srv1711671185=35, srv1658741159=29, srv1455442750=20, srv2110137983=61, srv851879123=94, srv393745905=68, srv567656518=78, srv1299933155=13, srv1833486111=44, srv1818291625=42, srv801602011=90, srv443868613=72, srv140694039=18, srv452570705=73, srv1173328186=9, srv1296444058=12, srv30976376=65, srv1143000408=8, srv1772683108=40, srv1665431057=30, srv1693546116=34, srv1476029536=21, srv195160449=53, srv100091796=0, srv646786044=84, srv1841541305=45, srv85694144=95, srv962765548=98, srv1565174562=25, srv33480024=66, srv1489889413=22, srv164106960=28, srv675048355=88, srv1322495483=14, srv1914494785=51, srv1756084493=38, srv1863571168=46, srv617367867=81, srv903895227=96, srv1866614417=47, srv2031706853=58, srv25323499=62, srv82998332=92, srv594057986=80, srv1539272226=23, srv2032172788=59, srv306421395=64, srv535732061=76, srv181838762=43, srv1073262092=6, srv1203049086=10, srv1101473436=7, srv1322682902=15, srv626739711=82, srv832875297=93, srv1691073220=33, srv648236756=85, srv1965815862=54, srv673568557=87, srv1671189252=31, srv6380927=83, srv437342945=70, srv49323549=74, srv581998937=79, srv1780323301=41, srv1025578333=2, srv2045136361=60, srv1539452449=24, srv1601458961=26, srv68734350=89, srv1887575737=49, srv1688659034=32, srv1071625643=5, srv1966521433=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-10 16:56:44,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-10 16:56:44,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-10 16:56:44,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-10 16:56:44,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-10 16:56:44,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-10 16:56:44,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-10 16:56:44,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-10 16:56:44,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-10 16:56:44,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-10 16:56:44,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-10 16:56:44,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-10 16:56:44,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-10 16:56:44,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-10 16:56:44,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-10 16:56:44,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-10 16:56:44,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-10 16:56:44,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-10 16:56:44,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-10 16:56:44,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-10 16:56:44,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-10 16:56:44,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-10 16:56:44,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-10 16:56:44,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-10 16:56:44,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-10 16:56:44,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-10 16:56:44,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-10 16:56:44,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-10 16:56:44,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-10 16:56:44,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-10 16:56:44,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-10 16:56:44,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-10 16:56:44,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-10 16:56:44,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-10 16:56:44,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-10 16:56:44,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-10 16:56:44,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-10 16:56:44,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-10 16:56:44,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-10 16:56:44,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-10 16:56:44,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-10 16:56:44,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-10 16:56:44,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-10 16:56:44,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-10 16:56:44,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-10 16:56:44,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-10 16:56:44,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-10 16:56:44,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-10 16:56:44,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-10 16:56:44,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-10 16:56:44,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-10 16:56:44,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-10 16:56:44,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-10 16:56:44,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-10 16:56:44,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-10 16:56:44,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-10 16:56:44,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-10 16:56:44,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-10 16:56:44,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-10 16:56:44,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-10 16:56:44,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-10 16:56:44,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-10 16:56:44,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-10 16:56:44,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-10 16:56:44,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-10 16:56:44,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-10 16:56:44,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-10 16:56:44,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-10 16:56:44,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-10 16:56:44,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-10 16:56:44,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-10 16:56:44,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-10 16:56:44,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-10 16:56:44,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-10 16:56:44,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-10 16:56:44,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-10 16:56:44,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-10 16:56:44,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-10 16:56:44,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-10 16:56:44,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-10 16:56:44,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-10 16:56:44,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-10 16:56:44,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-10 16:56:44,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-10 16:56:44,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-10 16:56:44,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-10 16:56:44,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-10 16:56:44,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-10 16:56:44,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-10 16:56:44,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-10 16:56:44,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-10 16:56:44,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-10 16:56:44,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-10 16:56:44,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-10 16:56:44,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-10 16:56:44,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-10 16:56:44,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-10 16:56:44,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-10 16:56:44,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-10 16:56:44,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-10 16:56:44,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-10 16:56:44,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-10 16:56:44,219 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-10 16:56:44,219 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-10 16:56:44,219 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-10 16:56:44,484 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 388 ms to try 5000 different iterations. Found a solution that moves 4702 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17418173732609302. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03981469736103574, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055253725542333515, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5224444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03981469736103574, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16933333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23616666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-10 16:56:44,506 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv981805548:81 , srv1965815862:82 , srv181838762:83 , srv195160449:83 , srv1966521433:83 , srv291927189:83 , srv567656518:83 , srv675048355:83 , srv1018654109:84 , srv1476029536:84 , srv1688659034:84 , srv1833486111:84 , srv1884575519:84 , srv306421395:84 , srv33480024:84 , srv338783135:84 , srv443868613:84 , srv6380927:84 , srv648236756:84 , srv1173328186:85 , srv122097009:85 , srv1296444058:85 , srv1299933155:85 , srv1455442750:85 , srv1763408484:85 , srv1780323301:85 , srv1863571168:85 , srv1903296531:85 , srv194297072:85 , srv2031706853:85 , srv538219435:85 , srv832875297:85 , srv1061117445:86 , srv1143000408:86 , srv1203049086:86 , srv1322682902:86 , srv1359208880:86 , srv1539272226:86 , srv1539452449:86 , srv1565174562:86 , srv1658741159:86 , srv1717976470:86 , srv1735830053:86 , srv1841541305:86 , srv2032172788:86 , srv2045136361:86 , srv2110137983:86 , srv421398282:86 , srv441834510:86 , srv594057986:86 , srv626739711:86 , srv673568557:86 , srv801602011:86 , srv85694144:86 , srv903895227:86 , srv962765548:86 , srv1025578333:87 , srv1073262092:87 , srv1427897215:87 , srv1489889413:87 , srv1601458961:87 , srv1665431057:87 , srv1671189252:87 , srv1693546116:87 , srv1711671185:87 , srv1866614417:87 , srv25323499:87 , srv452570705:87 , srv502404229:87 , srv646786044:87 , srv68734350:87 , srv819857485:87 , srv851879123:87 , srv100091796:88 , srv1029196969:88 , srv1071625643:88 , srv135222558:88 , srv1612038212:88 , srv164106960:88 , srv1691073220:88 , srv1818291625:88 , srv1914494785:88 , srv1993796595:88 , srv437342945:88 , srv535732061:88 , srv617367867:88 , srv82998332:88 , srv1101473436:89 , srv1322495483:89 , srv1990635220:89 , srv393745905:89 , srv30976376:90 , srv1887575737:91 , srv49323549:91 , srv581998937:91 , srv665030735:91 , srv1772683108:93 , srv1756084493:94 , srv95658054:94 , srv140694039:444 } 2023-04-10 16:56:44,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-10 16:56:44,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv338783135=67, srv1990635220=56, srv291927189=63, srv819857485=91, srv1427897215=19, srv1903296531=50, srv1029196969=3, srv1612038212=27, srv95658054=97, srv441834510=71, srv1717976470=36, srv1018654109=1, srv135222558=16, srv665030735=86, srv981805548=99, srv1061117445=4, srv1763408484=39, srv1993796595=57, srv502404229=75, srv122097009=11, srv1359208880=17, srv1884575519=48, srv538219435=77, srv194297072=52, srv421398282=69, srv1735830053=37, srv1711671185=35, srv1658741159=29, srv1455442750=20, srv2110137983=61, srv851879123=94, srv393745905=68, srv567656518=78, srv1299933155=13, srv1833486111=44, srv1818291625=42, srv801602011=90, srv443868613=72, srv140694039=18, srv452570705=73, srv1173328186=9, srv1296444058=12, srv30976376=65, srv1143000408=8, srv1772683108=40, srv1665431057=30, srv1693546116=34, srv1476029536=21, srv195160449=53, srv100091796=0, srv646786044=84, srv1841541305=45, srv85694144=95, srv962765548=98, srv1565174562=25, srv33480024=66, srv1489889413=22, srv164106960=28, srv675048355=88, srv1322495483=14, srv1914494785=51, srv1756084493=38, srv1863571168=46, srv617367867=81, srv903895227=96, srv1866614417=47, srv2031706853=58, srv25323499=62, srv82998332=92, srv594057986=80, srv1539272226=23, srv2032172788=59, srv306421395=64, srv535732061=76, srv181838762=43, srv1073262092=6, srv1203049086=10, srv1101473436=7, srv1322682902=15, srv626739711=82, srv832875297=93, srv1691073220=33, srv648236756=85, srv1965815862=54, srv673568557=87, srv1671189252=31, srv6380927=83, srv437342945=70, srv49323549=74, srv581998937=79, srv1780323301=41, srv1025578333=2, srv2045136361=60, srv1539452449=24, srv1601458961=26, srv68734350=89, srv1887575737=49, srv1688659034=32, srv1071625643=5, srv1966521433=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-10 16:56:44,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-10 16:56:44,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-10 16:56:44,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-10 16:56:44,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-10 16:56:44,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-10 16:56:44,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-10 16:56:44,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-10 16:56:44,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-10 16:56:44,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-10 16:56:44,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-10 16:56:44,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-10 16:56:44,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-10 16:56:44,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-10 16:56:44,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-10 16:56:44,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-10 16:56:44,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-10 16:56:44,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-10 16:56:44,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-10 16:56:44,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-10 16:56:44,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-10 16:56:44,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-10 16:56:44,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-10 16:56:44,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-10 16:56:44,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-10 16:56:44,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-10 16:56:44,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-10 16:56:44,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-10 16:56:44,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-10 16:56:44,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-10 16:56:44,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-10 16:56:44,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-10 16:56:44,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-10 16:56:44,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-10 16:56:44,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-10 16:56:44,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-10 16:56:44,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-10 16:56:44,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-10 16:56:44,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-10 16:56:44,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-10 16:56:44,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-10 16:56:44,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-10 16:56:44,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-10 16:56:44,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-10 16:56:44,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-10 16:56:44,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-10 16:56:44,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-10 16:56:44,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-10 16:56:44,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-10 16:56:44,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-10 16:56:44,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-10 16:56:44,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-10 16:56:44,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-10 16:56:44,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-10 16:56:44,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-10 16:56:44,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-10 16:56:44,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-10 16:56:44,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-10 16:56:44,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-10 16:56:44,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-10 16:56:44,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-10 16:56:44,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-10 16:56:44,660 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03981469736103574, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055253725542333515, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03981469736103574, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16933333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23616666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-10 16:56:44,660 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-10 16:56:44,660 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17414881321839193, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03981469736103574, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055253725542333515, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03981469736103574, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16933333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23616666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-10 16:56:44,933 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 373 ms to try 5000 different iterations. Found a solution that moves 2689 regions; Going from a computed imbalance of 0.17414881321839193 to a new imbalance of 0.014762160360724096. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01994067770313378); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030620225411725353, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29877777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01994067770313378); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-10 16:56:44,944 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv95658054:78 , srv1756084493:79 , srv1772683108:80 , srv1887575737:80 , srv30976376:80 , srv100091796:82 , srv1101473436:82 , srv49323549:82 , srv665030735:82 , srv135222558:83 , srv393745905:83 , srv851879123:83 , srv1025578333:84 , srv1071625643:84 , srv1427897215:84 , srv1601458961:84 , srv1665431057:84 , srv1691073220:84 , srv1693546116:84 , srv1914494785:84 , srv2045136361:84 , srv437342945:84 , srv502404229:84 , srv535732061:84 , srv581998937:84 , srv1143000408:85 , srv1203049086:85 , srv1322495483:85 , srv1455442750:85 , srv1612038212:85 , srv1818291625:85 , srv1841541305:85 , srv1866614417:85 , srv1903296531:85 , srv1990635220:85 , srv1993796595:85 , srv2032172788:85 , srv25323499:85 , srv452570705:85 , srv538219435:85 , srv626739711:85 , srv646786044:85 , srv68734350:85 , srv832875297:85 , srv1029196969:86 , srv1173328186:86 , srv122097009:86 , srv1296444058:86 , srv1299933155:86 , srv1322682902:86 , srv1539272226:86 , srv1565174562:86 , srv164106960:86 , srv1658741159:86 , srv1671189252:86 , srv1688659034:86 , srv1711671185:86 , srv1735830053:86 , srv1863571168:86 , srv194297072:86 , srv594057986:86 , srv617367867:86 , srv801602011:86 , srv819857485:86 , srv1073262092:87 , srv1359208880:87 , srv1476029536:87 , srv1489889413:87 , srv1539452449:87 , srv1717976470:87 , srv1780323301:87 , srv195160449:87 , srv33480024:87 , srv421398282:87 , srv673568557:87 , srv82998332:87 , srv85694144:87 , srv962765548:87 , srv1061117445:88 , srv1763408484:88 , srv1884575519:88 , srv1966521433:88 , srv2031706853:88 , srv2110137983:88 , srv443868613:88 , srv675048355:88 , srv903895227:88 , srv1018654109:89 , srv181838762:89 , srv1833486111:89 , srv306421395:89 , srv338783135:89 , srv6380927:89 , srv648236756:89 , srv981805548:89 , srv1965815862:91 , srv291927189:91 , srv441834510:94 , srv567656518:94 , srv140694039:507 } 2023-04-10 16:56:44,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-10 16:56:44,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv338783135=67, srv1990635220=56, srv291927189=63, srv819857485=91, srv1427897215=19, srv1903296531=50, srv1029196969=3, srv1612038212=27, srv95658054=97, srv441834510=71, srv1717976470=36, srv1018654109=1, srv135222558=16, srv665030735=86, srv981805548=99, srv1061117445=4, srv1763408484=39, srv1993796595=57, srv502404229=75, srv122097009=11, srv1359208880=17, srv1884575519=48, srv538219435=77, srv194297072=52, srv421398282=69, srv1735830053=37, srv1711671185=35, srv1658741159=29, srv1455442750=20, srv2110137983=61, srv851879123=94, srv393745905=68, srv567656518=78, srv1299933155=13, srv1833486111=44, srv1818291625=42, srv801602011=90, srv443868613=72, srv140694039=18, srv452570705=73, srv1173328186=9, srv1296444058=12, srv30976376=65, srv1143000408=8, srv1772683108=40, srv1665431057=30, srv1693546116=34, srv1476029536=21, srv195160449=53, srv100091796=0, srv646786044=84, srv1841541305=45, srv85694144=95, srv962765548=98, srv1565174562=25, srv33480024=66, srv1489889413=22, srv164106960=28, srv675048355=88, srv1322495483=14, srv1914494785=51, srv1756084493=38, srv1863571168=46, srv617367867=81, srv903895227=96, srv1866614417=47, srv2031706853=58, srv25323499=62, srv82998332=92, srv594057986=80, srv1539272226=23, srv2032172788=59, srv306421395=64, srv535732061=76, srv181838762=43, srv1073262092=6, srv1203049086=10, srv1101473436=7, srv1322682902=15, srv626739711=82, srv832875297=93, srv1691073220=33, srv648236756=85, srv1965815862=54, srv673568557=87, srv1671189252=31, srv6380927=83, srv437342945=70, srv49323549=74, srv581998937=79, srv1780323301=41, srv1025578333=2, srv2045136361=60, srv1539452449=24, srv1601458961=26, srv68734350=89, srv1887575737=49, srv1688659034=32, srv1071625643=5, srv1966521433=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-10 16:56:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-10 16:56:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-10 16:56:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-10 16:56:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-10 16:56:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-10 16:56:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-10 16:56:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-10 16:56:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-10 16:56:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-10 16:56:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-10 16:56:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-10 16:56:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-10 16:56:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-10 16:56:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-10 16:56:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-10 16:56:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-10 16:56:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-10 16:56:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-10 16:56:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-10 16:56:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-10 16:56:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-10 16:56:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-10 16:56:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-10 16:56:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-10 16:56:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-10 16:56:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-10 16:56:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-10 16:56:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-10 16:56:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-10 16:56:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-10 16:56:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-10 16:56:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-10 16:56:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-10 16:56:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-10 16:56:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-10 16:56:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-10 16:56:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-10 16:56:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-10 16:56:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-10 16:56:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-10 16:56:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-10 16:56:45,088 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01994067770313378); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030620225411725353, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01994067770313378); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-10 16:56:45,088 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-10 16:56:45,088 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014743331580288503, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01994067770313378); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030620225411725353, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01994067770313378); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0445, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-10 16:56:45,382 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 401 ms to try 5000 different iterations. Found a solution that moves 1056 regions; Going from a computed imbalance of 0.014743331580288503 to a new imbalance of 5.341323989472984E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005095423149695045); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006564879518897745); MoveCostFunction : (multiplier=7.0, imbalance=0.11733333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005095423149695045); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-10 16:56:45,384 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv441834510:78 , srv567656518:80 , srv82998332:82 , srv1029196969:83 , srv1489889413:83 , srv164106960:83 , srv1990635220:83 , srv2110137983:83 , srv581998937:83 , srv617367867:83 , srv1018654109:84 , srv1061117445:84 , srv1073262092:84 , srv1322495483:84 , srv1359208880:84 , srv1612038212:84 , srv1671189252:84 , srv1711671185:84 , srv1763408484:84 , srv1772683108:84 , srv1965815862:84 , srv291927189:84 , srv306421395:84 , srv338783135:84 , srv49323549:84 , srv6380927:84 , srv673568557:84 , srv819857485:84 , srv903895227:84 , srv1322682902:85 , srv1539452449:85 , srv1691073220:85 , srv1717976470:85 , srv1735830053:85 , srv1780323301:85 , srv1818291625:85 , srv1833486111:85 , srv1866614417:85 , srv1914494785:85 , srv1993796595:85 , srv25323499:85 , srv437342945:85 , srv443868613:85 , srv594057986:85 , srv646786044:85 , srv648236756:85 , srv665030735:85 , srv68734350:85 , srv85694144:85 , srv962765548:85 , srv1071625643:86 , srv1203049086:86 , srv1296444058:86 , srv1427897215:86 , srv1539272226:86 , srv1565174562:86 , srv1601458961:86 , srv1658741159:86 , srv1693546116:86 , srv1756084493:86 , srv181838762:86 , srv1841541305:86 , srv1863571168:86 , srv1884575519:86 , srv194297072:86 , srv1966521433:86 , srv2031706853:86 , srv2032172788:86 , srv393745905:86 , srv421398282:86 , srv452570705:86 , srv502404229:86 , srv535732061:86 , srv626739711:86 , srv801602011:86 , srv95658054:86 , srv1025578333:87 , srv1101473436:87 , srv1143000408:87 , srv122097009:87 , srv1299933155:87 , srv135222558:87 , srv1476029536:87 , srv1665431057:87 , srv1887575737:87 , srv1903296531:87 , srv33480024:87 , srv538219435:87 , srv675048355:87 , srv832875297:87 , srv981805548:87 , srv100091796:88 , srv1455442750:88 , srv1688659034:88 , srv195160449:88 , srv2045136361:88 , srv30976376:88 , srv851879123:88 , srv1173328186:89 , srv140694039:552 } 2023-04-10 16:56:45,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-10 16:56:45,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv338783135=67, srv1990635220=56, srv291927189=63, srv819857485=91, srv1427897215=19, srv1903296531=50, srv1029196969=3, srv1612038212=27, srv95658054=97, srv441834510=71, srv1717976470=36, srv1018654109=1, srv135222558=16, srv665030735=86, srv981805548=99, srv1061117445=4, srv1763408484=39, srv1993796595=57, srv502404229=75, srv122097009=11, srv1359208880=17, srv1884575519=48, srv538219435=77, srv194297072=52, srv421398282=69, srv1735830053=37, srv1711671185=35, srv1658741159=29, srv1455442750=20, srv2110137983=61, srv851879123=94, srv393745905=68, srv567656518=78, srv1299933155=13, srv1833486111=44, srv1818291625=42, srv801602011=90, srv443868613=72, srv140694039=18, srv452570705=73, srv1173328186=9, srv1296444058=12, srv30976376=65, srv1143000408=8, srv1772683108=40, srv1665431057=30, srv1693546116=34, srv1476029536=21, srv195160449=53, srv100091796=0, srv646786044=84, srv1841541305=45, srv85694144=95, srv962765548=98, srv1565174562=25, srv33480024=66, srv1489889413=22, srv164106960=28, srv675048355=88, srv1322495483=14, srv1914494785=51, srv1756084493=38, srv1863571168=46, srv617367867=81, srv903895227=96, srv1866614417=47, srv2031706853=58, srv25323499=62, srv82998332=92, srv594057986=80, srv1539272226=23, srv2032172788=59, srv306421395=64, srv535732061=76, srv181838762=43, srv1073262092=6, srv1203049086=10, srv1101473436=7, srv1322682902=15, srv626739711=82, srv832875297=93, srv1691073220=33, srv648236756=85, srv1965815862=54, srv673568557=87, srv1671189252=31, srv6380927=83, srv437342945=70, srv49323549=74, srv581998937=79, srv1780323301=41, srv1025578333=2, srv2045136361=60, srv1539452449=24, srv1601458961=26, srv68734350=89, srv1887575737=49, srv1688659034=32, srv1071625643=5, srv1966521433=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-10 16:56:45,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-10 16:56:45,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-10 16:56:45,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-10 16:56:45,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-10 16:56:45,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-10 16:56:45,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-10 16:56:45,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-10 16:56:45,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-10 16:56:45,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-10 16:56:45,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-10 16:56:45,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-10 16:56:45,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-10 16:56:45,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-10 16:56:45,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-10 16:56:45,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-10 16:56:45,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-10 16:56:45,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-10 16:56:45,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-10 16:56:45,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-10 16:56:45,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-10 16:56:45,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-10 16:56:45,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-10 16:56:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-10 16:56:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-10 16:56:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-10 16:56:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-10 16:56:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-10 16:56:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-10 16:56:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-10 16:56:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-10 16:56:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-10 16:56:45,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-10 16:56:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-10 16:56:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-10 16:56:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-10 16:56:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-10 16:56:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-10 16:56:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-10 16:56:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-10 16:56:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-10 16:56:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-10 16:56:45,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-10 16:56:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-10 16:56:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-10 16:56:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-10 16:56:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-10 16:56:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-10 16:56:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-10 16:56:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-10 16:56:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-10 16:56:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-10 16:56:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-10 16:56:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-10 16:56:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-10 16:56:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-10 16:56:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-10 16:56:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-10 16:56:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-10 16:56:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-10 16:56:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-10 16:56:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-10 16:56:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-10 16:56:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-10 16:56:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-10 16:56:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-10 16:56:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-10 16:56:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-10 16:56:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-10 16:56:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-10 16:56:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-10 16:56:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-10 16:56:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-10 16:56:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-10 16:56:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-10 16:56:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-10 16:56:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-10 16:56:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-10 16:56:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-10 16:56:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-10 16:56:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-10 16:56:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-10 16:56:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-10 16:56:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-10 16:56:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-10 16:56:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-10 16:56:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-10 16:56:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-10 16:56:45,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-10 16:56:45,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-10 16:56:45,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-10 16:56:45,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-10 16:56:45,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-10 16:56:45,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-10 16:56:45,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-10 16:56:45,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-10 16:56:45,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-10 16:56:45,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-10 16:56:45,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-10 16:56:45,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-10 16:56:45,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-10 16:56:45,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-10 16:56:45,541 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005095423149695045); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006564879518897745); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005095423149695045); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-10 16:56:45,542 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-10 16:56:45,542 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.267381289063958E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005095423149695045); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006564879518897745); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005095423149695045); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-10 16:56:45,828 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 382 ms to try 5000 different iterations. Found a solution that moves 217 regions; Going from a computed imbalance of 5.267381289063958E-4 to a new imbalance of 1.8272730105628357E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.02411111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-10 16:56:45,830 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1173328186:82 , srv1756084493:83 , srv1818291625:83 , srv2031706853:83 , srv421398282:83 , srv673568557:83 , srv100091796:84 , srv1025578333:84 , srv1061117445:84 , srv1071625643:84 , srv1073262092:84 , srv1101473436:84 , srv1143000408:84 , srv122097009:84 , srv1322495483:84 , srv135222558:84 , srv1455442750:84 , srv1476029536:84 , srv1539272226:84 , srv1539452449:84 , srv1565174562:84 , srv1658741159:84 , srv1665431057:84 , srv1688659034:84 , srv1717976470:84 , srv181838762:84 , srv1833486111:84 , srv1884575519:84 , srv1887575737:84 , srv195160449:84 , srv1993796595:84 , srv2032172788:84 , srv2045136361:84 , srv2110137983:84 , srv291927189:84 , srv30976376:84 , srv33480024:84 , srv393745905:84 , srv441834510:84 , srv452570705:84 , srv535732061:84 , srv581998937:84 , srv648236756:84 , srv665030735:84 , srv675048355:84 , srv801602011:84 , srv85694144:84 , srv903895227:84 , srv95658054:84 , srv1018654109:85 , srv1029196969:85 , srv1203049086:85 , srv1296444058:85 , srv1299933155:85 , srv1322682902:85 , srv1359208880:85 , srv1427897215:85 , srv1489889413:85 , srv1601458961:85 , srv1612038212:85 , srv164106960:85 , srv1671189252:85 , srv1691073220:85 , srv1693546116:85 , srv1711671185:85 , srv1735830053:85 , srv1763408484:85 , srv1772683108:85 , srv1780323301:85 , srv1841541305:85 , srv1863571168:85 , srv1866614417:85 , srv1903296531:85 , srv1914494785:85 , srv1965815862:85 , srv1966521433:85 , srv1990635220:85 , srv25323499:85 , srv306421395:85 , srv338783135:85 , srv437342945:85 , srv443868613:85 , srv49323549:85 , srv502404229:85 , srv538219435:85 , srv567656518:85 , srv594057986:85 , srv617367867:85 , srv626739711:85 , srv6380927:85 , srv646786044:85 , srv819857485:85 , srv82998332:85 , srv832875297:85 , srv851879123:85 , srv962765548:85 , srv981805548:85 , srv194297072:86 , srv68734350:86 , srv140694039:639 } 2023-04-10 16:56:45,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-10 16:56:45,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv338783135=67, srv1990635220=56, srv291927189=63, srv819857485=91, srv1427897215=19, srv1903296531=50, srv1029196969=3, srv1612038212=27, srv95658054=97, srv441834510=71, srv1717976470=36, srv1018654109=1, srv135222558=16, srv665030735=86, srv981805548=99, srv1061117445=4, srv1763408484=39, srv1993796595=57, srv502404229=75, srv122097009=11, srv1359208880=17, srv1884575519=48, srv538219435=77, srv194297072=52, srv421398282=69, srv1735830053=37, srv1711671185=35, srv1658741159=29, srv1455442750=20, srv2110137983=61, srv851879123=94, srv393745905=68, srv567656518=78, srv1299933155=13, srv1833486111=44, srv1818291625=42, srv801602011=90, srv443868613=72, srv140694039=18, srv452570705=73, srv1173328186=9, srv1296444058=12, srv30976376=65, srv1143000408=8, srv1772683108=40, srv1665431057=30, srv1693546116=34, srv1476029536=21, srv195160449=53, srv100091796=0, srv646786044=84, srv1841541305=45, srv85694144=95, srv962765548=98, srv1565174562=25, srv33480024=66, srv1489889413=22, srv164106960=28, srv675048355=88, srv1322495483=14, srv1914494785=51, srv1756084493=38, srv1863571168=46, srv617367867=81, srv903895227=96, srv1866614417=47, srv2031706853=58, srv25323499=62, srv82998332=92, srv594057986=80, srv1539272226=23, srv2032172788=59, srv306421395=64, srv535732061=76, srv181838762=43, srv1073262092=6, srv1203049086=10, srv1101473436=7, srv1322682902=15, srv626739711=82, srv832875297=93, srv1691073220=33, srv648236756=85, srv1965815862=54, srv673568557=87, srv1671189252=31, srv6380927=83, srv437342945=70, srv49323549=74, srv581998937=79, srv1780323301=41, srv1025578333=2, srv2045136361=60, srv1539452449=24, srv1601458961=26, srv68734350=89, srv1887575737=49, srv1688659034=32, srv1071625643=5, srv1966521433=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-10 16:56:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-10 16:56:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-10 16:56:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-10 16:56:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-10 16:56:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-10 16:56:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-10 16:56:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-10 16:56:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-10 16:56:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-10 16:56:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-10 16:56:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-10 16:56:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-10 16:56:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-10 16:56:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-10 16:56:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-10 16:56:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-10 16:56:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-10 16:56:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-10 16:56:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-10 16:56:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-10 16:56:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-10 16:56:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-10 16:56:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-10 16:56:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-10 16:56:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-10 16:56:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-10 16:56:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-10 16:56:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-10 16:56:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-10 16:56:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-10 16:56:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-10 16:56:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-10 16:56:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-10 16:56:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-10 16:56:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-10 16:56:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-10 16:56:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-10 16:56:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-10 16:56:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-10 16:56:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-10 16:56:45,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-10 16:56:45,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-10 16:56:45,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-10 16:56:45,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-10 16:56:45,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-10 16:56:45,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-10 16:56:45,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-10 16:56:45,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-10 16:56:45,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-10 16:56:45,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-10 16:56:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-10 16:56:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-10 16:56:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-10 16:56:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-10 16:56:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-10 16:56:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-10 16:56:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-10 16:56:45,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-10 16:56:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-10 16:56:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-10 16:56:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-10 16:56:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-10 16:56:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-10 16:56:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-10 16:56:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-10 16:56:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-10 16:56:45,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-10 16:56:45,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-10 16:56:45,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-10 16:56:45,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-10 16:56:45,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-10 16:56:45,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-10 16:56:45,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-10 16:56:45,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-10 16:56:45,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-10 16:56:45,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-10 16:56:45,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-10 16:56:45,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-10 16:56:45,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-10 16:56:45,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-10 16:56:45,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-10 16:56:45,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-10 16:56:45,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-10 16:56:45,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-10 16:56:45,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-10 16:56:45,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-10 16:56:45,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-10 16:56:45,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-10 16:56:45,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-10 16:56:45,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-10 16:56:45,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-10 16:56:45,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-10 16:56:45,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-10 16:56:45,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-10 16:56:45,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-10 16:56:45,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-10 16:56:45,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-10 16:56:45,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-10 16:56:45,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-10 16:56:45,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-10 16:56:45,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-10 16:56:45,980 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.675326362942016E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-10 16:56:45,980 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1173328186:82 , srv1756084493:83 , srv1818291625:83 , srv2031706853:83 , srv421398282:83 , srv673568557:83 , srv100091796:84 , srv1025578333:84 , srv1061117445:84 , srv1071625643:84 , srv1073262092:84 , srv1101473436:84 , srv1143000408:84 , srv122097009:84 , srv1322495483:84 , srv135222558:84 , srv1455442750:84 , srv1476029536:84 , srv1539272226:84 , srv1539452449:84 , srv1565174562:84 , srv1658741159:84 , srv1665431057:84 , srv1688659034:84 , srv1717976470:84 , srv181838762:84 , srv1833486111:84 , srv1884575519:84 , srv1887575737:84 , srv195160449:84 , srv1993796595:84 , srv2032172788:84 , srv2045136361:84 , srv2110137983:84 , srv291927189:84 , srv30976376:84 , srv33480024:84 , srv393745905:84 , srv441834510:84 , srv452570705:84 , srv535732061:84 , srv581998937:84 , srv648236756:84 , srv665030735:84 , srv675048355:84 , srv801602011:84 , srv85694144:84 , srv903895227:84 , srv95658054:84 , srv1018654109:85 , srv1029196969:85 , srv1203049086:85 , srv1296444058:85 , srv1299933155:85 , srv1322682902:85 , srv1359208880:85 , srv1427897215:85 , srv1489889413:85 , srv1601458961:85 , srv1612038212:85 , srv164106960:85 , srv1671189252:85 , srv1691073220:85 , srv1693546116:85 , srv1711671185:85 , srv1735830053:85 , srv1763408484:85 , srv1772683108:85 , srv1780323301:85 , srv1841541305:85 , srv1863571168:85 , srv1866614417:85 , srv1903296531:85 , srv1914494785:85 , srv1965815862:85 , srv1966521433:85 , srv1990635220:85 , srv25323499:85 , srv306421395:85 , srv338783135:85 , srv437342945:85 , srv443868613:85 , srv49323549:85 , srv502404229:85 , srv538219435:85 , srv567656518:85 , srv594057986:85 , srv617367867:85 , srv626739711:85 , srv6380927:85 , srv646786044:85 , srv819857485:85 , srv82998332:85 , srv832875297:85 , srv851879123:85 , srv962765548:85 , srv981805548:85 , srv194297072:86 , srv68734350:86 , srv140694039:639 } 2023-04-10 16:56:46,010 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=415 (was 416), ProcessCount=169 (was 169), AvailableMemoryMB=895 (was 1370) 2023-04-10 16:56:46,018 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=415, ProcessCount=169, AvailableMemoryMB=895 2023-04-10 16:56:46,018 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-10 16:56:46,019 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-04-10 16:56:46,029 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1089013250:84 , srv1231390654:84 , srv1280786161:84 , srv1289678323:84 , srv1298385812:84 , srv1312153449:84 , srv1507926417:84 , srv1530958850:84 , srv1542204156:84 , srv1632832001:84 , srv1658248298:84 , srv1756618348:84 , srv198439268:84 , srv2036314843:84 , srv2045774531:84 , srv2051548626:84 , srv2080465570:84 , srv270537266:84 , srv298132621:84 , srv320673027:84 , srv418090829:84 , srv467497016:84 , srv5201025:84 , srv700169937:84 , srv837160176:84 , srv910372949:84 , srv915837002:84 , srv963916489:84 , srv988863213:84 , srv1731019133:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-04-10 16:56:46,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-10 16:56:46,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1312153449=5, srv1530958850=7, srv2045774531=15, srv5201025=23, srv1289678323=3, srv700169937=24, srv2051548626=16, srv910372949=26, srv988863213=29, srv1632832001=9, srv963916489=28, srv2080465570=17, srv418090829=21, srv1280786161=2, srv1298385812=4, srv1542204156=8, srv915837002=27, srv270537266=18, srv198439268=13, srv1089013250=0, srv1731019133=11, srv2036314843=14, srv467497016=22, srv320673027=20, srv1507926417=6, srv837160176=25, srv1231390654=1, srv1756618348=12, srv1658248298=10, srv298132621=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-10 16:56:46,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-10 16:56:46,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-10 16:56:46,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-10 16:56:46,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-10 16:56:46,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-10 16:56:46,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-10 16:56:46,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-10 16:56:46,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-10 16:56:46,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-10 16:56:46,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-10 16:56:46,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-10 16:56:46,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-10 16:56:46,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-10 16:56:46,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-10 16:56:46,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-10 16:56:46,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-10 16:56:46,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-10 16:56:46,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-10 16:56:46,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-10 16:56:46,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-10 16:56:46,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-10 16:56:46,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-10 16:56:46,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-10 16:56:46,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-10 16:56:46,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-10 16:56:46,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-10 16:56:46,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-10 16:56:46,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-10 16:56:46,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-10 16:56:46,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-10 16:56:46,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-10 16:56:46,055 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1027341381355824, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-10 16:56:46,055 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912516755178967, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1027341381355824, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-04-10 16:58:46,044 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6269249 different iterations. Found a solution that moves 1847 regions; Going from a computed imbalance of 0.9912516755178967 to a new imbalance of 4.3109901406398434E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6840740740740741, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-10 16:58:46,052 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1089013250:90 , srv1231390654:90 , srv1280786161:90 , srv1289678323:90 , srv1298385812:90 , srv1312153449:90 , srv1507926417:90 , srv1530958850:90 , srv1542204156:90 , srv1632832001:90 , srv1658248298:90 , srv1731019133:90 , srv1756618348:90 , srv198439268:90 , srv2036314843:90 , srv2045774531:90 , srv2051548626:90 , srv2080465570:90 , srv270537266:90 , srv298132621:90 , srv320673027:90 , srv418090829:90 , srv467497016:90 , srv5201025:90 , srv700169937:90 , srv837160176:90 , srv910372949:90 , srv915837002:90 , srv963916489:90 , srv988863213:90 } 2023-04-10 16:58:46,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-10 16:58:46,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1312153449=5, srv1530958850=7, srv2045774531=15, srv5201025=23, srv1289678323=3, srv700169937=24, srv2051548626=16, srv910372949=26, srv988863213=29, srv1632832001=9, srv963916489=28, srv2080465570=17, srv418090829=21, srv1280786161=2, srv1298385812=4, srv1542204156=8, srv915837002=27, srv270537266=18, srv198439268=13, srv1089013250=0, srv1731019133=11, srv2036314843=14, srv467497016=22, srv320673027=20, srv1507926417=6, srv837160176=25, srv1231390654=1, srv1756618348=12, srv1658248298=10, srv298132621=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-10 16:58:46,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-10 16:58:46,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-10 16:58:46,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-10 16:58:46,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-10 16:58:46,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-10 16:58:46,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-10 16:58:46,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-10 16:58:46,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-10 16:58:46,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-10 16:58:46,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-10 16:58:46,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-10 16:58:46,076 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-10 16:58:46,076 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1089013250:90 , srv1231390654:90 , srv1280786161:90 , srv1289678323:90 , srv1298385812:90 , srv1312153449:90 , srv1507926417:90 , srv1530958850:90 , srv1542204156:90 , srv1632832001:90 , srv1658248298:90 , srv1731019133:90 , srv1756618348:90 , srv198439268:90 , srv2036314843:90 , srv2045774531:90 , srv2051548626:90 , srv2080465570:90 , srv270537266:90 , srv298132621:90 , srv320673027:90 , srv418090829:90 , srv467497016:90 , srv5201025:90 , srv700169937:90 , srv837160176:90 , srv910372949:90 , srv915837002:90 , srv963916489:90 , srv988863213:90 } 2023-04-10 16:58:46,091 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=290 (was 415), ProcessCount=168 (was 169), AvailableMemoryMB=1149 (was 895) - AvailableMemoryMB LEAK? -