2023-12-11 00:53:14,323 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-11 00:53:14,665 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-11 00:53:14,791 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-11 00:53:14,851 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-11 00:53:14,858 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-11 00:53:14,874 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=342, ProcessCount=170, AvailableMemoryMB=6398 2023-12-11 00:53:14,877 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-11 00:53:14,877 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-11 00:53:15,018 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1008148428:84 , srv1010282853:84 , srv1021618300:84 , srv1029218459:84 , srv1042014776:84 , srv1057658570:84 , srv1059497770:84 , srv1067618620:84 , srv1071568414:84 , srv108368752:84 , srv1140236232:84 , srv1169078288:84 , srv1208173337:84 , srv1224448212:84 , srv1235199241:84 , srv125450408:84 , srv126160226:84 , srv1271454950:84 , srv1303934475:84 , srv1328131721:84 , srv1351360933:84 , srv1384848915:84 , srv1420299581:84 , srv1433167232:84 , srv1439971635:84 , srv1462900947:84 , srv1466346119:84 , srv1467765381:84 , srv147941529:84 , srv1499314165:84 , srv1502362214:84 , srv1538230693:84 , srv1565752927:84 , srv1575273616:84 , srv1648294978:84 , srv16526508:84 , srv1654785496:84 , srv1658845181:84 , srv166752913:84 , srv1702631617:84 , srv171055465:84 , srv1720536586:84 , srv1722795465:84 , srv1753314485:84 , srv1861800355:84 , srv1883158905:84 , srv1907273312:84 , srv1922477706:84 , srv1934343501:84 , srv1938905959:84 , srv1949312139:84 , srv1951933542:84 , srv1968281171:84 , srv1970112783:84 , srv1970192980:84 , srv1978546178:84 , srv1990191741:84 , srv2049349864:84 , srv2064146738:84 , srv2103790011:84 , srv2115684445:84 , srv2127397713:84 , srv2142789137:84 , srv253413795:84 , srv258550389:84 , srv327857985:84 , srv351982008:84 , srv371767703:84 , srv376519403:84 , srv385329953:84 , srv390743502:84 , srv396652324:84 , srv439403245:84 , srv47160111:84 , srv533658013:84 , srv541639429:84 , srv561011612:84 , srv585867771:84 , srv589721851:84 , srv599588211:84 , srv620631527:84 , srv623913415:84 , srv635001321:84 , srv664729029:84 , srv711598537:84 , srv714125432:84 , srv789412637:84 , srv792679900:84 , srv797242826:84 , srv843158002:84 , srv858212769:84 , srv863087946:84 , srv869453249:84 , srv884368103:84 , srv898421303:84 , srv900231537:84 , srv904892643:84 , srv985517647:84 , srv996897650:84 , srv1309473173:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-11 00:53:15,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-11 00:53:15,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1140236232=10, srv1938905959=50, srv396652324=72, srv711598537=85, srv390743502=71, srv1970192980=55, srv1720536586=42, srv1433167232=24, srv126160226=16, srv1949312139=51, srv385329953=70, srv797242826=89, srv47160111=74, srv1029218459=3, srv1753314485=44, srv1565752927=33, srv1883158905=46, srv258550389=65, srv623913415=82, srv585867771=78, srv884368103=94, srv1008148428=0, srv1658845181=38, srv1722795465=43, srv869453249=93, srv1462900947=26, srv1309473173=19, srv1970112783=54, srv371767703=68, srv789412637=87, srv900231537=96, srv541639429=76, srv1648294978=35, srv599588211=80, srv792679900=88, srv1575273616=34, srv1654785496=37, srv1990191741=57, srv2064146738=59, srv533658013=75, srv1499314165=30, srv327857985=66, srv351982008=67, srv589721851=79, srv1067618620=7, srv171055465=41, srv439403245=73, srv714125432=86, srv147941529=29, srv253413795=64, srv1208173337=12, srv1271454950=17, srv1538230693=32, srv16526508=36, srv1224448212=13, srv1702631617=40, srv985517647=98, srv1466346119=27, srv1439971635=25, srv996897650=99, srv858212769=91, srv1021618300=2, srv1169078288=11, srv1934343501=49, srv1235199241=14, srv620631527=81, srv1907273312=47, srv1059497770=6, srv108368752=9, srv1384848915=22, srv898421303=95, srv1978546178=56, srv664729029=84, srv2049349864=58, srv2115684445=61, srv1303934475=18, srv1502362214=31, srv2142789137=63, srv2103790011=60, srv1057658570=5, srv843158002=90, srv1071568414=8, srv1328131721=20, srv561011612=77, srv125450408=15, srv1010282853=1, srv1351360933=21, srv1042014776=4, srv166752913=39, srv376519403=69, srv863087946=92, srv1467765381=28, srv904892643=97, srv1968281171=53, srv2127397713=62, srv1861800355=45, srv1951933542=52, srv1922477706=48, srv1420299581=23, srv635001321=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-11 00:53:15,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-11 00:53:15,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-11 00:53:15,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-11 00:53:15,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-11 00:53:15,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-11 00:53:15,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-11 00:53:15,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-11 00:53:15,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-11 00:53:15,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-11 00:53:15,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-11 00:53:15,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-11 00:53:15,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-11 00:53:15,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-11 00:53:15,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-11 00:53:15,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-11 00:53:15,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-11 00:53:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-11 00:53:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-11 00:53:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-11 00:53:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-11 00:53:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-11 00:53:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-11 00:53:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-11 00:53:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-11 00:53:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-11 00:53:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-11 00:53:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-11 00:53:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-11 00:53:15,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-11 00:53:15,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-11 00:53:15,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-11 00:53:15,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-11 00:53:15,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-11 00:53:15,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-11 00:53:15,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-11 00:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-11 00:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-11 00:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-11 00:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-11 00:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-11 00:53:15,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-11 00:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-11 00:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-11 00:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-11 00:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-11 00:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-11 00:53:15,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-11 00:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-11 00:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-11 00:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-11 00:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-11 00:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-11 00:53:15,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-11 00:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-11 00:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-11 00:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-11 00:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-11 00:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-11 00:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-11 00:53:15,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-11 00:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-11 00:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-11 00:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-11 00:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-11 00:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-11 00:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-11 00:53:15,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-11 00:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-11 00:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-11 00:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-11 00:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-11 00:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-11 00:53:15,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-11 00:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-11 00:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-11 00:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-11 00:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-11 00:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-11 00:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-11 00:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-11 00:53:15,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-11 00:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-11 00:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-11 00:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-11 00:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-11 00:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-11 00:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-11 00:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-11 00:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-11 00:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-11 00:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-11 00:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-11 00:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-11 00:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-11 00:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-11 00:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-11 00:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-11 00:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-11 00:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-11 00:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-11 00:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-11 00:53:15,399 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:53:15,400 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-11 00:53:15,400 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-11 00:53:15,828 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 630 ms to try 5000 different iterations. Found a solution that moves 4699 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17780108457251062. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04081291433854103, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05324276822372876, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5221111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04081291433854103, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17341666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23558333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:53:15,872 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1439971635:79 , srv108368752:81 , srv147941529:81 , srv16526508:81 , srv1968281171:81 , srv439403245:81 , srv1502362214:82 , srv47160111:82 , srv714125432:82 , srv843158002:82 , srv1224448212:83 , srv1467765381:83 , srv1565752927:83 , srv1654785496:83 , srv1753314485:83 , srv1938905959:83 , srv2103790011:83 , srv664729029:83 , srv797242826:83 , srv863087946:83 , srv1071568414:84 , srv1140236232:84 , srv1208173337:84 , srv1303934475:84 , srv1328131721:84 , srv1499314165:84 , srv1720536586:84 , srv1883158905:84 , srv1934343501:84 , srv2142789137:84 , srv620631527:84 , srv900231537:84 , srv904892643:84 , srv1057658570:85 , srv1067618620:85 , srv1235199241:85 , srv171055465:85 , srv371767703:85 , srv376519403:85 , srv396652324:85 , srv533658013:85 , srv541639429:85 , srv898421303:85 , srv985517647:85 , srv996897650:85 , srv1169078288:86 , srv1575273616:86 , srv1922477706:86 , srv1951933542:86 , srv1978546178:86 , srv1990191741:86 , srv327857985:86 , srv351982008:86 , srv585867771:86 , srv599588211:86 , srv635001321:86 , srv869453249:86 , srv884368103:86 , srv125450408:87 , srv1384848915:87 , srv1462900947:87 , srv1658845181:87 , srv1970112783:87 , srv711598537:87 , srv858212769:87 , srv1059497770:88 , srv1433167232:88 , srv1648294978:88 , srv1702631617:88 , srv1949312139:88 , srv2064146738:88 , srv253413795:88 , srv561011612:88 , srv623913415:88 , srv792679900:88 , srv1008148428:89 , srv1466346119:89 , srv1722795465:89 , srv2127397713:89 , srv385329953:89 , srv1029218459:90 , srv1271454950:90 , srv1538230693:90 , srv166752913:90 , srv1861800355:90 , srv2049349864:90 , srv789412637:90 , srv1010282853:91 , srv1021618300:91 , srv1351360933:91 , srv1970192980:91 , srv390743502:91 , srv1042014776:92 , srv126160226:92 , srv589721851:92 , srv1907273312:93 , srv2115684445:93 , srv258550389:94 , srv1420299581:101 , srv1309473173:452 } 2023-12-11 00:53:15,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-11 00:53:15,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1140236232=10, srv1938905959=50, srv396652324=72, srv711598537=85, srv390743502=71, srv1970192980=55, srv1720536586=42, srv1433167232=24, srv126160226=16, srv1949312139=51, srv385329953=70, srv797242826=89, srv47160111=74, srv1029218459=3, srv1753314485=44, srv1565752927=33, srv1883158905=46, srv258550389=65, srv623913415=82, srv585867771=78, srv884368103=94, srv1008148428=0, srv1658845181=38, srv1722795465=43, srv869453249=93, srv1462900947=26, srv1309473173=19, srv1970112783=54, srv371767703=68, srv789412637=87, srv900231537=96, srv541639429=76, srv1648294978=35, srv599588211=80, srv792679900=88, srv1575273616=34, srv1654785496=37, srv1990191741=57, srv2064146738=59, srv533658013=75, srv1499314165=30, srv327857985=66, srv351982008=67, srv589721851=79, srv1067618620=7, srv171055465=41, srv439403245=73, srv714125432=86, srv147941529=29, srv253413795=64, srv1208173337=12, srv1271454950=17, srv1538230693=32, srv16526508=36, srv1224448212=13, srv1702631617=40, srv985517647=98, srv1466346119=27, srv1439971635=25, srv996897650=99, srv858212769=91, srv1021618300=2, srv1169078288=11, srv1934343501=49, srv1235199241=14, srv620631527=81, srv1907273312=47, srv1059497770=6, srv108368752=9, srv1384848915=22, srv898421303=95, srv1978546178=56, srv664729029=84, srv2049349864=58, srv2115684445=61, srv1303934475=18, srv1502362214=31, srv2142789137=63, srv2103790011=60, srv1057658570=5, srv843158002=90, srv1071568414=8, srv1328131721=20, srv561011612=77, srv125450408=15, srv1010282853=1, srv1351360933=21, srv1042014776=4, srv166752913=39, srv376519403=69, srv863087946=92, srv1467765381=28, srv904892643=97, srv1968281171=53, srv2127397713=62, srv1861800355=45, srv1951933542=52, srv1922477706=48, srv1420299581=23, srv635001321=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-11 00:53:15,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-11 00:53:15,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-11 00:53:15,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-11 00:53:15,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-11 00:53:15,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-11 00:53:15,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-11 00:53:15,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-11 00:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-11 00:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-11 00:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-11 00:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-11 00:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-11 00:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-11 00:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-11 00:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-11 00:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-11 00:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-11 00:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-11 00:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-11 00:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-11 00:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-11 00:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-11 00:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-11 00:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-11 00:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-11 00:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-11 00:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-11 00:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-11 00:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-11 00:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-11 00:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-11 00:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-11 00:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-11 00:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-11 00:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-11 00:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-11 00:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-11 00:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-11 00:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-11 00:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-11 00:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-11 00:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-11 00:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-11 00:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-11 00:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-11 00:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-11 00:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-11 00:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-11 00:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-11 00:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-11 00:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-11 00:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-11 00:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-11 00:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-11 00:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-11 00:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-11 00:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-11 00:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-11 00:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-11 00:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-11 00:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-11 00:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-11 00:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-11 00:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-11 00:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-11 00:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-11 00:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-11 00:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-11 00:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-11 00:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-11 00:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-11 00:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-11 00:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-11 00:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-11 00:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-11 00:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-11 00:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-11 00:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-11 00:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-11 00:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-11 00:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-11 00:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-11 00:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-11 00:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-11 00:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-11 00:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-11 00:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-11 00:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-11 00:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-11 00:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-11 00:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-11 00:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-11 00:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-11 00:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-11 00:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-11 00:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-11 00:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-11 00:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-11 00:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-11 00:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-11 00:53:15,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-11 00:53:16,116 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04081291433854103, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05324276822372876, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04081291433854103, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17341666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23558333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:53:16,117 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-11 00:53:16,117 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17776818147125853, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04081291433854103, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05324276822372876, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04081291433854103, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17341666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23558333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-11 00:53:16,414 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 454 ms to try 5000 different iterations. Found a solution that moves 2607 regions; Going from a computed imbalance of 0.17776818147125853 to a new imbalance of 0.01353048579307176. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022720562670757515); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029151872149686196, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2896666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022720562670757515); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04075, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:53:16,428 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1420299581:75 , srv1042014776:78 , srv258550389:79 , srv126160226:80 , srv589721851:80 , srv1010282853:81 , srv1021618300:81 , srv166752913:81 , srv1907273312:81 , srv2049349864:81 , srv1059497770:82 , srv1271454950:82 , srv1351360933:82 , srv1538230693:82 , srv1702631617:82 , srv1722795465:82 , srv1861800355:82 , srv1970192980:82 , srv2064146738:82 , srv2115684445:82 , srv1008148428:83 , srv1029218459:83 , srv1433167232:83 , srv1466346119:83 , srv390743502:83 , srv789412637:83 , srv792679900:83 , srv125450408:84 , srv1462900947:84 , srv1648294978:84 , srv1922477706:84 , srv1949312139:84 , srv1970112783:84 , srv2127397713:84 , srv327857985:84 , srv585867771:84 , srv623913415:84 , srv1169078288:85 , srv1235199241:85 , srv1575273616:85 , srv1658845181:85 , srv1951933542:85 , srv1978546178:85 , srv351982008:85 , srv561011612:85 , srv635001321:85 , srv869453249:85 , srv1384848915:86 , srv1720536586:86 , srv253413795:86 , srv385329953:86 , srv396652324:86 , srv541639429:86 , srv599588211:86 , srv858212769:86 , srv1067618620:87 , srv1208173337:87 , srv1303934475:87 , srv1328131721:87 , srv1753314485:87 , srv1990191741:87 , srv2142789137:87 , srv371767703:87 , srv376519403:87 , srv664729029:87 , srv711598537:87 , srv898421303:87 , srv900231537:87 , srv985517647:87 , srv996897650:87 , srv1140236232:88 , srv1502362214:88 , srv1654785496:88 , srv171055465:88 , srv1934343501:88 , srv1938905959:88 , srv714125432:88 , srv884368103:88 , srv1057658570:89 , srv1467765381:89 , srv1499314165:89 , srv1565752927:89 , srv1883158905:89 , srv533658013:89 , srv797242826:89 , srv863087946:89 , srv1224448212:90 , srv16526508:90 , srv2103790011:90 , srv439403245:90 , srv47160111:90 , srv620631527:90 , srv1071568414:91 , srv108368752:91 , srv147941529:91 , srv843158002:91 , srv904892643:91 , srv1439971635:92 , srv1968281171:92 , srv1309473173:524 } 2023-12-11 00:53:16,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-11 00:53:16,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1140236232=10, srv1938905959=50, srv396652324=72, srv711598537=85, srv390743502=71, srv1970192980=55, srv1720536586=42, srv1433167232=24, srv126160226=16, srv1949312139=51, srv385329953=70, srv797242826=89, srv47160111=74, srv1029218459=3, srv1753314485=44, srv1565752927=33, srv1883158905=46, srv258550389=65, srv623913415=82, srv585867771=78, srv884368103=94, srv1008148428=0, srv1658845181=38, srv1722795465=43, srv869453249=93, srv1462900947=26, srv1309473173=19, srv1970112783=54, srv371767703=68, srv789412637=87, srv900231537=96, srv541639429=76, srv1648294978=35, srv599588211=80, srv792679900=88, srv1575273616=34, srv1654785496=37, srv1990191741=57, srv2064146738=59, srv533658013=75, srv1499314165=30, srv327857985=66, srv351982008=67, srv589721851=79, srv1067618620=7, srv171055465=41, srv439403245=73, srv714125432=86, srv147941529=29, srv253413795=64, srv1208173337=12, srv1271454950=17, srv1538230693=32, srv16526508=36, srv1224448212=13, srv1702631617=40, srv985517647=98, srv1466346119=27, srv1439971635=25, srv996897650=99, srv858212769=91, srv1021618300=2, srv1169078288=11, srv1934343501=49, srv1235199241=14, srv620631527=81, srv1907273312=47, srv1059497770=6, srv108368752=9, srv1384848915=22, srv898421303=95, srv1978546178=56, srv664729029=84, srv2049349864=58, srv2115684445=61, srv1303934475=18, srv1502362214=31, srv2142789137=63, srv2103790011=60, srv1057658570=5, srv843158002=90, srv1071568414=8, srv1328131721=20, srv561011612=77, srv125450408=15, srv1010282853=1, srv1351360933=21, srv1042014776=4, srv166752913=39, srv376519403=69, srv863087946=92, srv1467765381=28, srv904892643=97, srv1968281171=53, srv2127397713=62, srv1861800355=45, srv1951933542=52, srv1922477706=48, srv1420299581=23, srv635001321=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-11 00:53:16,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-11 00:53:16,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-11 00:53:16,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-11 00:53:16,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-11 00:53:16,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-11 00:53:16,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-11 00:53:16,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-11 00:53:16,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-11 00:53:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-11 00:53:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-11 00:53:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-11 00:53:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-11 00:53:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-11 00:53:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-11 00:53:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-11 00:53:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-11 00:53:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-11 00:53:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-11 00:53:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-11 00:53:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-11 00:53:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-11 00:53:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-11 00:53:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-11 00:53:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-11 00:53:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-11 00:53:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-11 00:53:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-11 00:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-11 00:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-11 00:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-11 00:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-11 00:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-11 00:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-11 00:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-11 00:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-11 00:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-11 00:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-11 00:53:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-11 00:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-11 00:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-11 00:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-11 00:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-11 00:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-11 00:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-11 00:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-11 00:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-11 00:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-11 00:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-11 00:53:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-11 00:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-11 00:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-11 00:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-11 00:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-11 00:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-11 00:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-11 00:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-11 00:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-11 00:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-11 00:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-11 00:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-11 00:53:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-11 00:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-11 00:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-11 00:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-11 00:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-11 00:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-11 00:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-11 00:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-11 00:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-11 00:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-11 00:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-11 00:53:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-11 00:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-11 00:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-11 00:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-11 00:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-11 00:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-11 00:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-11 00:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-11 00:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-11 00:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-11 00:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-11 00:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-11 00:53:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-11 00:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-11 00:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-11 00:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-11 00:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-11 00:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-11 00:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-11 00:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-11 00:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-11 00:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-11 00:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-11 00:53:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-11 00:53:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-11 00:53:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-11 00:53:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-11 00:53:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-11 00:53:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-11 00:53:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-11 00:53:16,646 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022720562670757515); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029151872149686196, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022720562670757515); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04075, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:53:16,646 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-11 00:53:16,647 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013512231188908281, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022720562670757515); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029151872149686196, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022720562670757515); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04075, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-11 00:53:16,926 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 429 ms to try 5000 different iterations. Found a solution that moves 918 regions; Going from a computed imbalance of 0.013512231188908281 to a new imbalance of 5.592093960345E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00698813417189378); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00865863393605529); MoveCostFunction : (multiplier=7.0, imbalance=0.102, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00698813417189378); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:53:16,928 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv904892643:81 , srv1420299581:82 , srv1057658570:83 , srv2115684445:83 , srv253413795:83 , srv385329953:83 , srv390743502:83 , srv711598537:83 , srv843158002:83 , srv1029218459:84 , srv1071568414:84 , srv1224448212:84 , srv1883158905:84 , srv1907273312:84 , srv1968281171:84 , srv1970192980:84 , srv1990191741:84 , srv2103790011:84 , srv2127397713:84 , srv533658013:84 , srv620631527:84 , srv789412637:84 , srv884368103:84 , srv1008148428:85 , srv1067618620:85 , srv108368752:85 , srv1140236232:85 , srv126160226:85 , srv1271454950:85 , srv1351360933:85 , srv1384848915:85 , srv1466346119:85 , srv1467765381:85 , srv147941529:85 , srv1499314165:85 , srv1538230693:85 , srv1565752927:85 , srv1658845181:85 , srv171055465:85 , srv1861800355:85 , srv1934343501:85 , srv1949312139:85 , srv258550389:85 , srv371767703:85 , srv47160111:85 , srv589721851:85 , srv599588211:85 , srv797242826:85 , srv858212769:85 , srv898421303:85 , srv985517647:85 , srv1010282853:86 , srv1021618300:86 , srv1169078288:86 , srv1328131721:86 , srv1439971635:86 , srv1575273616:86 , srv1648294978:86 , srv1654785496:86 , srv166752913:86 , srv1722795465:86 , srv1951933542:86 , srv1970112783:86 , srv1978546178:86 , srv2049349864:86 , srv2142789137:86 , srv376519403:86 , srv396652324:86 , srv439403245:86 , srv541639429:86 , srv561011612:86 , srv623913415:86 , srv635001321:86 , srv792679900:86 , srv863087946:86 , srv996897650:86 , srv1042014776:87 , srv125450408:87 , srv1303934475:87 , srv1433167232:87 , srv1462900947:87 , srv1502362214:87 , srv16526508:87 , srv1702631617:87 , srv1720536586:87 , srv1753314485:87 , srv1922477706:87 , srv1938905959:87 , srv351982008:87 , srv585867771:87 , srv664729029:87 , srv714125432:87 , srv869453249:87 , srv900231537:87 , srv1059497770:88 , srv1208173337:88 , srv1235199241:88 , srv2064146738:88 , srv327857985:88 , srv1309473173:544 } 2023-12-11 00:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-11 00:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1140236232=10, srv1938905959=50, srv396652324=72, srv711598537=85, srv390743502=71, srv1970192980=55, srv1720536586=42, srv1433167232=24, srv126160226=16, srv1949312139=51, srv385329953=70, srv797242826=89, srv47160111=74, srv1029218459=3, srv1753314485=44, srv1565752927=33, srv1883158905=46, srv258550389=65, srv623913415=82, srv585867771=78, srv884368103=94, srv1008148428=0, srv1658845181=38, srv1722795465=43, srv869453249=93, srv1462900947=26, srv1309473173=19, srv1970112783=54, srv371767703=68, srv789412637=87, srv900231537=96, srv541639429=76, srv1648294978=35, srv599588211=80, srv792679900=88, srv1575273616=34, srv1654785496=37, srv1990191741=57, srv2064146738=59, srv533658013=75, srv1499314165=30, srv327857985=66, srv351982008=67, srv589721851=79, srv1067618620=7, srv171055465=41, srv439403245=73, srv714125432=86, srv147941529=29, srv253413795=64, srv1208173337=12, srv1271454950=17, srv1538230693=32, srv16526508=36, srv1224448212=13, srv1702631617=40, srv985517647=98, srv1466346119=27, srv1439971635=25, srv996897650=99, srv858212769=91, srv1021618300=2, srv1169078288=11, srv1934343501=49, srv1235199241=14, srv620631527=81, srv1907273312=47, srv1059497770=6, srv108368752=9, srv1384848915=22, srv898421303=95, srv1978546178=56, srv664729029=84, srv2049349864=58, srv2115684445=61, srv1303934475=18, srv1502362214=31, srv2142789137=63, srv2103790011=60, srv1057658570=5, srv843158002=90, srv1071568414=8, srv1328131721=20, srv561011612=77, srv125450408=15, srv1010282853=1, srv1351360933=21, srv1042014776=4, srv166752913=39, srv376519403=69, srv863087946=92, srv1467765381=28, srv904892643=97, srv1968281171=53, srv2127397713=62, srv1861800355=45, srv1951933542=52, srv1922477706=48, srv1420299581=23, srv635001321=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-11 00:53:16,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-11 00:53:16,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-11 00:53:16,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-11 00:53:16,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-11 00:53:16,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-11 00:53:16,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-11 00:53:16,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-11 00:53:17,071 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00698813417189378); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00865863393605529); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00698813417189378); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:53:17,071 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-11 00:53:17,072 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.527814226466699E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00698813417189378); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00865863393605529); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00698813417189378); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-11 00:53:17,538 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 576 ms to try 5000 different iterations. Found a solution that moves 217 regions; Going from a computed imbalance of 5.527814226466699E-4 to a new imbalance of 4.1230590272588454E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.02411111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:53:17,539 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1071568414:83 , srv1208173337:83 , srv1907273312:83 , srv533658013:83 , srv561011612:83 , srv1010282853:84 , srv1021618300:84 , srv1029218459:84 , srv1059497770:84 , srv1235199241:84 , srv125450408:84 , srv1303934475:84 , srv1351360933:84 , srv1384848915:84 , srv1420299581:84 , srv1433167232:84 , srv1462900947:84 , srv1466346119:84 , srv1499314165:84 , srv1648294978:84 , srv16526508:84 , srv1654785496:84 , srv1722795465:84 , srv1938905959:84 , srv2064146738:84 , srv258550389:84 , srv327857985:84 , srv351982008:84 , srv376519403:84 , srv439403245:84 , srv620631527:84 , srv623913415:84 , srv858212769:84 , srv863087946:84 , srv869453249:84 , srv884368103:84 , srv900231537:84 , srv996897650:84 , srv1008148428:85 , srv1042014776:85 , srv1057658570:85 , srv1067618620:85 , srv108368752:85 , srv1140236232:85 , srv1169078288:85 , srv1224448212:85 , srv1271454950:85 , srv1328131721:85 , srv1439971635:85 , srv1467765381:85 , srv147941529:85 , srv1502362214:85 , srv1538230693:85 , srv1565752927:85 , srv1575273616:85 , srv1658845181:85 , srv166752913:85 , srv1702631617:85 , srv171055465:85 , srv1720536586:85 , srv1753314485:85 , srv1861800355:85 , srv1883158905:85 , srv1922477706:85 , srv1934343501:85 , srv1949312139:85 , srv1951933542:85 , srv1968281171:85 , srv1970112783:85 , srv1970192980:85 , srv1978546178:85 , srv1990191741:85 , srv2049349864:85 , srv2103790011:85 , srv2115684445:85 , srv2127397713:85 , srv2142789137:85 , srv253413795:85 , srv371767703:85 , srv385329953:85 , srv390743502:85 , srv396652324:85 , srv47160111:85 , srv541639429:85 , srv585867771:85 , srv589721851:85 , srv599588211:85 , srv635001321:85 , srv711598537:85 , srv714125432:85 , srv789412637:85 , srv797242826:85 , srv898421303:85 , srv985517647:85 , srv126160226:86 , srv664729029:86 , srv792679900:86 , srv843158002:86 , srv904892643:87 , srv1309473173:622 } 2023-12-11 00:53:17,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-11 00:53:17,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1140236232=10, srv1938905959=50, srv396652324=72, srv711598537=85, srv390743502=71, srv1970192980=55, srv1720536586=42, srv1433167232=24, srv126160226=16, srv1949312139=51, srv385329953=70, srv797242826=89, srv47160111=74, srv1029218459=3, srv1753314485=44, srv1565752927=33, srv1883158905=46, srv258550389=65, srv623913415=82, srv585867771=78, srv884368103=94, srv1008148428=0, srv1658845181=38, srv1722795465=43, srv869453249=93, srv1462900947=26, srv1309473173=19, srv1970112783=54, srv371767703=68, srv789412637=87, srv900231537=96, srv541639429=76, srv1648294978=35, srv599588211=80, srv792679900=88, srv1575273616=34, srv1654785496=37, srv1990191741=57, srv2064146738=59, srv533658013=75, srv1499314165=30, srv327857985=66, srv351982008=67, srv589721851=79, srv1067618620=7, srv171055465=41, srv439403245=73, srv714125432=86, srv147941529=29, srv253413795=64, srv1208173337=12, srv1271454950=17, srv1538230693=32, srv16526508=36, srv1224448212=13, srv1702631617=40, srv985517647=98, srv1466346119=27, srv1439971635=25, srv996897650=99, srv858212769=91, srv1021618300=2, srv1169078288=11, srv1934343501=49, srv1235199241=14, srv620631527=81, srv1907273312=47, srv1059497770=6, srv108368752=9, srv1384848915=22, srv898421303=95, srv1978546178=56, srv664729029=84, srv2049349864=58, srv2115684445=61, srv1303934475=18, srv1502362214=31, srv2142789137=63, srv2103790011=60, srv1057658570=5, srv843158002=90, srv1071568414=8, srv1328131721=20, srv561011612=77, srv125450408=15, srv1010282853=1, srv1351360933=21, srv1042014776=4, srv166752913=39, srv376519403=69, srv863087946=92, srv1467765381=28, srv904892643=97, srv1968281171=53, srv2127397713=62, srv1861800355=45, srv1951933542=52, srv1922477706=48, srv1420299581=23, srv635001321=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-11 00:53:17,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-11 00:53:17,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-11 00:53:17,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-11 00:53:17,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-11 00:53:17,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-11 00:53:17,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-11 00:53:17,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-11 00:53:17,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-11 00:53:17,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-11 00:53:17,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-11 00:53:17,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-11 00:53:17,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-11 00:53:17,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-11 00:53:17,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-11 00:53:17,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-11 00:53:17,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-11 00:53:17,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-11 00:53:17,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-11 00:53:17,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-11 00:53:17,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-11 00:53:17,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-11 00:53:17,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-11 00:53:17,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-11 00:53:17,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-11 00:53:17,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-11 00:53:17,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-11 00:53:17,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-11 00:53:17,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-11 00:53:17,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-11 00:53:17,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-11 00:53:17,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-11 00:53:17,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-11 00:53:17,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-11 00:53:17,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-11 00:53:17,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-11 00:53:17,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-11 00:53:17,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-11 00:53:17,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-11 00:53:17,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-11 00:53:17,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-11 00:53:17,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-11 00:53:17,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-11 00:53:17,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-11 00:53:17,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-11 00:53:17,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-11 00:53:17,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-11 00:53:17,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-11 00:53:17,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-11 00:53:17,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-11 00:53:17,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-11 00:53:17,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-11 00:53:17,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-11 00:53:17,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-11 00:53:17,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-11 00:53:17,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-11 00:53:17,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-11 00:53:17,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-11 00:53:17,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-11 00:53:17,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-11 00:53:17,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-11 00:53:17,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-11 00:53:17,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-11 00:53:17,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-11 00:53:17,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-11 00:53:17,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-11 00:53:17,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-11 00:53:17,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-11 00:53:17,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-11 00:53:17,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-11 00:53:17,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-11 00:53:17,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-11 00:53:17,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-11 00:53:17,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-11 00:53:17,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-11 00:53:17,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-11 00:53:17,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-11 00:53:17,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-11 00:53:17,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-11 00:53:17,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-11 00:53:17,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-11 00:53:17,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-11 00:53:17,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-11 00:53:17,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-11 00:53:17,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-11 00:53:17,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-11 00:53:17,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-11 00:53:17,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-11 00:53:17,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-11 00:53:17,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-11 00:53:17,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-11 00:53:17,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-11 00:53:17,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-11 00:53:17,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-11 00:53:17,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-11 00:53:17,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-11 00:53:17,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-11 00:53:17,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-11 00:53:17,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-11 00:53:17,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-11 00:53:17,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-11 00:53:17,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-11 00:53:17,691 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:53:17,691 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-11 00:53:17,692 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.971112379638026E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-11 00:53:17,966 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 379 ms to try 5000 different iterations. Found a solution that moves 23 regions; Going from a computed imbalance of 3.971112379638026E-5 to a new imbalance of 7.234288738508196E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0025555555555555557); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:53:17,967 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv126160226:83 , srv171055465:83 , srv2115684445:83 , srv385329953:83 , srv664729029:83 , srv792679900:83 , srv904892643:83 , srv1008148428:84 , srv1010282853:84 , srv1021618300:84 , srv1042014776:84 , srv1057658570:84 , srv1059497770:84 , srv1067618620:84 , srv1071568414:84 , srv108368752:84 , srv1140236232:84 , srv1169078288:84 , srv1208173337:84 , srv1224448212:84 , srv1235199241:84 , srv125450408:84 , srv1271454950:84 , srv1303934475:84 , srv1328131721:84 , srv1351360933:84 , srv1384848915:84 , srv1420299581:84 , srv1433167232:84 , srv1439971635:84 , srv1462900947:84 , srv1467765381:84 , srv147941529:84 , srv1499314165:84 , srv1502362214:84 , srv1538230693:84 , srv1565752927:84 , srv1575273616:84 , srv1648294978:84 , srv16526508:84 , srv1658845181:84 , srv166752913:84 , srv1702631617:84 , srv1720536586:84 , srv1753314485:84 , srv1861800355:84 , srv1883158905:84 , srv1922477706:84 , srv1934343501:84 , srv1938905959:84 , srv1949312139:84 , srv1951933542:84 , srv1968281171:84 , srv1970112783:84 , srv1970192980:84 , srv1978546178:84 , srv1990191741:84 , srv2049349864:84 , srv2064146738:84 , srv2103790011:84 , srv2127397713:84 , srv2142789137:84 , srv253413795:84 , srv258550389:84 , srv327857985:84 , srv351982008:84 , srv371767703:84 , srv376519403:84 , srv390743502:84 , srv396652324:84 , srv47160111:84 , srv541639429:84 , srv561011612:84 , srv585867771:84 , srv589721851:84 , srv599588211:84 , srv620631527:84 , srv623913415:84 , srv635001321:84 , srv711598537:84 , srv714125432:84 , srv789412637:84 , srv797242826:84 , srv843158002:84 , srv858212769:84 , srv863087946:84 , srv869453249:84 , srv884368103:84 , srv898421303:84 , srv900231537:84 , srv985517647:84 , srv996897650:84 , srv1029218459:85 , srv1466346119:85 , srv1654785496:85 , srv1722795465:85 , srv1907273312:85 , srv439403245:85 , srv533658013:85 , srv1309473173:684 } 2023-12-11 00:53:17,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-11 00:53:17,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1140236232=10, srv1938905959=50, srv396652324=72, srv711598537=85, srv390743502=71, srv1970192980=55, srv1720536586=42, srv1433167232=24, srv126160226=16, srv1949312139=51, srv385329953=70, srv797242826=89, srv47160111=74, srv1029218459=3, srv1753314485=44, srv1565752927=33, srv1883158905=46, srv258550389=65, srv623913415=82, srv585867771=78, srv884368103=94, srv1008148428=0, srv1658845181=38, srv1722795465=43, srv869453249=93, srv1462900947=26, srv1309473173=19, srv1970112783=54, srv371767703=68, srv789412637=87, srv900231537=96, srv541639429=76, srv1648294978=35, srv599588211=80, srv792679900=88, srv1575273616=34, srv1654785496=37, srv1990191741=57, srv2064146738=59, srv533658013=75, srv1499314165=30, srv327857985=66, srv351982008=67, srv589721851=79, srv1067618620=7, srv171055465=41, srv439403245=73, srv714125432=86, srv147941529=29, srv253413795=64, srv1208173337=12, srv1271454950=17, srv1538230693=32, srv16526508=36, srv1224448212=13, srv1702631617=40, srv985517647=98, srv1466346119=27, srv1439971635=25, srv996897650=99, srv858212769=91, srv1021618300=2, srv1169078288=11, srv1934343501=49, srv1235199241=14, srv620631527=81, srv1907273312=47, srv1059497770=6, srv108368752=9, srv1384848915=22, srv898421303=95, srv1978546178=56, srv664729029=84, srv2049349864=58, srv2115684445=61, srv1303934475=18, srv1502362214=31, srv2142789137=63, srv2103790011=60, srv1057658570=5, srv843158002=90, srv1071568414=8, srv1328131721=20, srv561011612=77, srv125450408=15, srv1010282853=1, srv1351360933=21, srv1042014776=4, srv166752913=39, srv376519403=69, srv863087946=92, srv1467765381=28, srv904892643=97, srv1968281171=53, srv2127397713=62, srv1861800355=45, srv1951933542=52, srv1922477706=48, srv1420299581=23, srv635001321=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-11 00:53:17,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-11 00:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-11 00:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-11 00:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-11 00:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-11 00:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-11 00:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-11 00:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-11 00:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-11 00:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-11 00:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-11 00:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-11 00:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-11 00:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-11 00:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-11 00:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-11 00:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-11 00:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-11 00:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-11 00:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-11 00:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-11 00:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-11 00:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-11 00:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-11 00:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-11 00:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-11 00:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-11 00:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-11 00:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-11 00:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-11 00:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-11 00:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-11 00:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-11 00:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-11 00:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-11 00:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-11 00:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-11 00:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-11 00:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-11 00:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-11 00:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-11 00:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-11 00:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-11 00:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-11 00:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-11 00:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-11 00:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-11 00:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-11 00:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-11 00:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-11 00:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-11 00:53:17,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-11 00:53:17,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-11 00:53:17,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-11 00:53:17,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-11 00:53:17,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-11 00:53:17,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-11 00:53:18,112 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:53:18,112 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv126160226:83 , srv171055465:83 , srv2115684445:83 , srv385329953:83 , srv664729029:83 , srv792679900:83 , srv904892643:83 , srv1008148428:84 , srv1010282853:84 , srv1021618300:84 , srv1042014776:84 , srv1057658570:84 , srv1059497770:84 , srv1067618620:84 , srv1071568414:84 , srv108368752:84 , srv1140236232:84 , srv1169078288:84 , srv1208173337:84 , srv1224448212:84 , srv1235199241:84 , srv125450408:84 , srv1271454950:84 , srv1303934475:84 , srv1328131721:84 , srv1351360933:84 , srv1384848915:84 , srv1420299581:84 , srv1433167232:84 , srv1439971635:84 , srv1462900947:84 , srv1467765381:84 , srv147941529:84 , srv1499314165:84 , srv1502362214:84 , srv1538230693:84 , srv1565752927:84 , srv1575273616:84 , srv1648294978:84 , srv16526508:84 , srv1658845181:84 , srv166752913:84 , srv1702631617:84 , srv1720536586:84 , srv1753314485:84 , srv1861800355:84 , srv1883158905:84 , srv1922477706:84 , srv1934343501:84 , srv1938905959:84 , srv1949312139:84 , srv1951933542:84 , srv1968281171:84 , srv1970112783:84 , srv1970192980:84 , srv1978546178:84 , srv1990191741:84 , srv2049349864:84 , srv2064146738:84 , srv2103790011:84 , srv2127397713:84 , srv2142789137:84 , srv253413795:84 , srv258550389:84 , srv327857985:84 , srv351982008:84 , srv371767703:84 , srv376519403:84 , srv390743502:84 , srv396652324:84 , srv47160111:84 , srv541639429:84 , srv561011612:84 , srv585867771:84 , srv589721851:84 , srv599588211:84 , srv620631527:84 , srv623913415:84 , srv635001321:84 , srv711598537:84 , srv714125432:84 , srv789412637:84 , srv797242826:84 , srv843158002:84 , srv858212769:84 , srv863087946:84 , srv869453249:84 , srv884368103:84 , srv898421303:84 , srv900231537:84 , srv985517647:84 , srv996897650:84 , srv1029218459:85 , srv1466346119:85 , srv1654785496:85 , srv1722795465:85 , srv1907273312:85 , srv439403245:85 , srv533658013:85 , srv1309473173:684 } 2023-12-11 00:53:18,139 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=401 (was 342) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=5503 (was 6398) 2023-12-11 00:53:18,150 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=401, ProcessCount=170, AvailableMemoryMB=5502 2023-12-11 00:53:18,151 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-11 00:53:18,151 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-11 00:53:18,165 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1005366084:84 , srv1065064770:84 , srv1111733200:84 , srv1178886101:84 , srv1283542189:84 , srv1342528932:84 , srv1410064360:84 , srv1543001784:84 , srv1581214914:84 , srv1596204221:84 , srv1633487227:84 , srv1695542040:84 , srv1773145006:84 , srv184007348:84 , srv1841811536:84 , srv1907541488:84 , srv2035782577:84 , srv2047833377:84 , srv2105152662:84 , srv323427774:84 , srv396237519:84 , srv406675334:84 , srv654690112:84 , srv683163385:84 , srv780363891:84 , srv837968198:84 , srv870846229:84 , srv874248588:84 , srv893988440:84 , srv820003945:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-11 00:53:18,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-11 00:53:18,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1695542040=11, srv820003945=25, srv1111733200=2, srv1342528932=5, srv2035782577=16, srv1773145006=12, srv323427774=19, srv2105152662=18, srv1841811536=14, srv2047833377=17, srv396237519=20, srv780363891=24, srv654690112=22, srv406675334=21, srv874248588=28, srv837968198=26, srv1065064770=1, srv683163385=23, srv1283542189=4, srv1633487227=10, srv1543001784=7, srv1907541488=15, srv1410064360=6, srv1596204221=9, srv1178886101=3, srv870846229=27, srv184007348=13, srv1005366084=0, srv1581214914=8, srv893988440=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-11 00:53:18,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-11 00:53:18,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-11 00:53:18,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-11 00:53:18,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-11 00:53:18,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-11 00:53:18,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-11 00:53:18,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-11 00:53:18,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-11 00:53:18,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-11 00:53:18,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-11 00:53:18,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-11 00:53:18,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-11 00:53:18,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-11 00:53:18,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-11 00:53:18,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-11 00:53:18,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-11 00:53:18,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-11 00:53:18,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-11 00:53:18,209 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1083533399491137, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:53:18,209 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912534461100397, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1083533399491137, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-11 00:55:18,188 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6148559 different iterations. Found a solution that moves 1849 regions; Going from a computed imbalance of 0.9912534461100397 to a new imbalance of 4.3156582404131396E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6848148148148148, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:55:18,195 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1005366084:90 , srv1065064770:90 , srv1111733200:90 , srv1178886101:90 , srv1283542189:90 , srv1342528932:90 , srv1410064360:90 , srv1543001784:90 , srv1581214914:90 , srv1596204221:90 , srv1633487227:90 , srv1695542040:90 , srv1773145006:90 , srv184007348:90 , srv1841811536:90 , srv1907541488:90 , srv2035782577:90 , srv2047833377:90 , srv2105152662:90 , srv323427774:90 , srv396237519:90 , srv406675334:90 , srv654690112:90 , srv683163385:90 , srv780363891:90 , srv820003945:90 , srv837968198:90 , srv870846229:90 , srv874248588:90 , srv893988440:90 } 2023-12-11 00:55:18,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-11 00:55:18,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1695542040=11, srv820003945=25, srv1111733200=2, srv1342528932=5, srv2035782577=16, srv1773145006=12, srv323427774=19, srv2105152662=18, srv1841811536=14, srv2047833377=17, srv396237519=20, srv780363891=24, srv654690112=22, srv406675334=21, srv874248588=28, srv837968198=26, srv1065064770=1, srv683163385=23, srv1283542189=4, srv1633487227=10, srv1543001784=7, srv1907541488=15, srv1410064360=6, srv1596204221=9, srv1178886101=3, srv870846229=27, srv184007348=13, srv1005366084=0, srv1581214914=8, srv893988440=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-11 00:55:18,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-11 00:55:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-11 00:55:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-11 00:55:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-11 00:55:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-11 00:55:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-11 00:55:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-11 00:55:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-11 00:55:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-11 00:55:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-11 00:55:18,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-11 00:55:18,214 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-11 00:55:18,214 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1005366084:90 , srv1065064770:90 , srv1111733200:90 , srv1178886101:90 , srv1283542189:90 , srv1342528932:90 , srv1410064360:90 , srv1543001784:90 , srv1581214914:90 , srv1596204221:90 , srv1633487227:90 , srv1695542040:90 , srv1773145006:90 , srv184007348:90 , srv1841811536:90 , srv1907541488:90 , srv2035782577:90 , srv2047833377:90 , srv2105152662:90 , srv323427774:90 , srv396237519:90 , srv406675334:90 , srv654690112:90 , srv683163385:90 , srv780363891:90 , srv820003945:90 , srv837968198:90 , srv870846229:90 , srv874248588:90 , srv893988440:90 } 2023-12-11 00:55:18,226 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=212 (was 401), ProcessCount=168 (was 170), AvailableMemoryMB=5122 (was 5502)