2023-08-07 13:53:14,774 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-07 13:53:15,102 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-07 13:53:15,219 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-07 13:53:15,293 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-07 13:53:15,301 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-07 13:53:15,320 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=284, ProcessCount=174, AvailableMemoryMB=3406 2023-08-07 13:53:15,323 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-07 13:53:15,323 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-07 13:53:15,471 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1012633449:84 , srv1033476424:84 , srv1052367495:84 , srv1062801435:84 , srv1083478858:84 , srv1092234546:84 , srv1094708566:84 , srv110135646:84 , srv1118227721:84 , srv1127644970:84 , srv1131750004:84 , srv1135165098:84 , srv1172359170:84 , srv1195608128:84 , srv1202711863:84 , srv1224987328:84 , srv1253375993:84 , srv1272286185:84 , srv1286264611:84 , srv1297922342:84 , srv1314302088:84 , srv1321358964:84 , srv1378746570:84 , srv1389037701:84 , srv1428177451:84 , srv1465228820:84 , srv1469399910:84 , srv1479042501:84 , srv1489557885:84 , srv1594293913:84 , srv1598975481:84 , srv1641576971:84 , srv1683108487:84 , srv1731159213:84 , srv1733863686:84 , srv1735940010:84 , srv1736213305:84 , srv1749364058:84 , srv175022303:84 , srv175219432:84 , srv1759233737:84 , srv1790963242:84 , srv179687780:84 , srv1823856869:84 , srv1866462693:84 , srv1945272266:84 , srv1947600258:84 , srv1951131263:84 , srv2013680364:84 , srv2014157401:84 , srv2030957120:84 , srv2039509502:84 , srv205818251:84 , srv2065333489:84 , srv2068728778:84 , srv2070650608:84 , srv2083247464:84 , srv2093195589:84 , srv2116201054:84 , srv2141118044:84 , srv218247140:84 , srv226885252:84 , srv268018004:84 , srv304863471:84 , srv30859695:84 , srv309543465:84 , srv323133853:84 , srv357487835:84 , srv362015878:84 , srv418979864:84 , srv424113286:84 , srv424720750:84 , srv436430590:84 , srv440304999:84 , srv461979205:84 , srv477534705:84 , srv493962020:84 , srv53021258:84 , srv597781971:84 , srv630319677:84 , srv644037916:84 , srv650005157:84 , srv651464475:84 , srv677302791:84 , srv696448729:84 , srv713080285:84 , srv730861495:84 , srv739382439:84 , srv744878033:84 , srv76778478:84 , srv77122411:84 , srv779286202:84 , srv798014095:84 , srv807300751:84 , srv816043509:84 , srv818002073:84 , srv881811220:84 , srv931546925:84 , srv941450114:84 , srv1081059137:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-07 13:53:15,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-07 13:53:15,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1736213305=37, srv2013680364=49, srv1378746570=23, srv424113286=71, srv1321358964=22, srv1428177451=25, srv2068728778=55, srv77122411=91, srv744878033=89, srv2014157401=50, srv1594293913=30, srv1081059137=4, srv1195608128=14, srv2030957120=51, srv1749364058=38, srv2070650608=56, srv1735940010=36, srv175022303=39, srv630319677=80, srv1733863686=35, srv1790963242=42, srv818002073=96, srv175219432=40, srv1286264611=19, srv1052367495=2, srv2083247464=57, srv1947600258=47, srv807300751=94, srv1131750004=11, srv2039509502=52, srv1598975481=31, srv362015878=69, srv597781971=79, srv1951131263=48, srv1945272266=46, srv1314302088=21, srv309543465=66, srv1135165098=12, srv436430590=73, srv677302791=84, srv931546925=98, srv713080285=86, srv1202711863=15, srv226885252=62, srv418979864=70, srv798014095=93, srv941450114=99, srv179687780=43, srv218247140=61, srv1489557885=29, srv1297922342=20, srv816043509=95, srv1479042501=28, srv1272286185=18, srv1389037701=24, srv739382439=88, srv110135646=8, srv2116201054=59, srv30859695=65, srv53021258=78, srv304863471=64, srv1641576971=32, srv1092234546=6, srv1172359170=13, srv1127644970=10, srv1118227721=9, srv2093195589=58, srv1759233737=41, srv1823856869=44, srv268018004=63, srv1866462693=45, srv76778478=90, srv2141118044=60, srv1094708566=7, srv1062801435=3, srv461979205=75, srv424720750=72, srv440304999=74, srv2065333489=54, srv779286202=92, srv881811220=97, srv696448729=85, srv1224987328=16, srv1469399910=27, srv1033476424=1, srv730861495=87, srv357487835=68, srv650005157=82, srv1253375993=17, srv644037916=81, srv1683108487=33, srv493962020=77, srv1731159213=34, srv651464475=83, srv205818251=53, srv1083478858=5, srv477534705=76, srv1012633449=0, srv1465228820=26, srv323133853=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-07 13:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-07 13:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-07 13:53:15,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-07 13:53:15,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-07 13:53:15,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-07 13:53:15,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-07 13:53:15,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-07 13:53:15,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-07 13:53:15,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-07 13:53:15,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-07 13:53:15,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-07 13:53:15,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-07 13:53:15,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-07 13:53:15,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-07 13:53:15,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-07 13:53:15,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-07 13:53:15,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-07 13:53:15,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-07 13:53:15,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-07 13:53:15,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-07 13:53:15,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-07 13:53:15,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-07 13:53:15,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-07 13:53:15,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-07 13:53:15,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-07 13:53:15,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-07 13:53:15,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-07 13:53:15,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-07 13:53:15,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-07 13:53:15,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-07 13:53:15,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-07 13:53:15,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-07 13:53:15,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-07 13:53:15,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-07 13:53:15,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-07 13:53:15,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-07 13:53:15,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-07 13:53:15,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-07 13:53:15,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-07 13:53:15,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-07 13:53:15,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-07 13:53:15,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-07 13:53:15,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-07 13:53:15,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-07 13:53:15,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-07 13:53:15,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-07 13:53:15,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-07 13:53:15,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-07 13:53:15,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-07 13:53:15,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-07 13:53:15,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-07 13:53:15,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-07 13:53:15,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-07 13:53:15,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-07 13:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-07 13:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-07 13:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-07 13:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-07 13:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-07 13:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-07 13:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-07 13:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-07 13:53:15,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-07 13:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-07 13:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-07 13:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-07 13:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-07 13:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-07 13:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-07 13:53:15,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-07 13:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-07 13:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-07 13:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-07 13:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-07 13:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-07 13:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-07 13:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-07 13:53:15,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-07 13:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-07 13:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-07 13:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-07 13:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-07 13:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-07 13:53:15,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-07 13:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-07 13:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-07 13:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-07 13:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-07 13:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-07 13:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-07 13:53:15,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-07 13:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-07 13:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-07 13:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-07 13:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-07 13:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-07 13:53:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-07 13:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-07 13:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-07 13:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-07 13:53:15,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-07 13:53:15,769 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:53:15,769 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-07 13:53:15,770 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-07 13:53:16,176 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 575 ms to try 5000 different iterations. Found a solution that moves 4698 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17984595211720023. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03928561468447832, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054154038932104895, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.522, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03928561468447832, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17433333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24916666666666668, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:53:16,203 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1052367495:82 , srv931546925:82 , srv1314302088:83 , srv1321358964:83 , srv1594293913:83 , srv175022303:83 , srv1759233737:83 , srv1945272266:83 , srv2014157401:83 , srv436430590:83 , srv818002073:83 , srv1012633449:84 , srv1135165098:84 , srv1489557885:84 , srv1733863686:84 , srv1866462693:84 , srv1951131263:84 , srv2039509502:84 , srv2141118044:84 , srv362015878:84 , srv744878033:84 , srv941450114:84 , srv1062801435:85 , srv1094708566:85 , srv1202711863:85 , srv1224987328:85 , srv1286264611:85 , srv1297922342:85 , srv1428177451:85 , srv1479042501:85 , srv1736213305:85 , srv1749364058:85 , srv2070650608:85 , srv218247140:85 , srv357487835:85 , srv440304999:85 , srv461979205:85 , srv644037916:85 , srv807300751:85 , srv1118227721:86 , srv1389037701:86 , srv1465228820:86 , srv1641576971:86 , srv1683108487:86 , srv1947600258:86 , srv2013680364:86 , srv2065333489:86 , srv2083247464:86 , srv226885252:86 , srv30859695:86 , srv418979864:86 , srv696448729:86 , srv713080285:86 , srv77122411:86 , srv816043509:86 , srv881811220:86 , srv1083478858:87 , srv1131750004:87 , srv1172359170:87 , srv1253375993:87 , srv1598975481:87 , srv2030957120:87 , srv268018004:87 , srv304863471:87 , srv677302791:87 , srv730861495:87 , srv739382439:87 , srv798014095:87 , srv1092234546:88 , srv1195608128:88 , srv1272286185:88 , srv1469399910:88 , srv1735940010:88 , srv175219432:88 , srv1790963242:88 , srv179687780:88 , srv1823856869:88 , srv2093195589:88 , srv323133853:88 , srv53021258:88 , srv630319677:88 , srv651464475:88 , srv779286202:88 , srv1033476424:89 , srv110135646:89 , srv1127644970:89 , srv1731159213:89 , srv205818251:89 , srv2068728778:89 , srv424720750:89 , srv650005157:89 , srv424113286:90 , srv477534705:90 , srv2116201054:92 , srv597781971:92 , srv1378746570:93 , srv309543465:93 , srv493962020:94 , srv76778478:99 , srv1081059137:439 } 2023-08-07 13:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-07 13:53:16,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1736213305=37, srv2013680364=49, srv1378746570=23, srv424113286=71, srv1321358964=22, srv1428177451=25, srv2068728778=55, srv77122411=91, srv744878033=89, srv2014157401=50, srv1594293913=30, srv1081059137=4, srv1195608128=14, srv2030957120=51, srv1749364058=38, srv2070650608=56, srv1735940010=36, srv175022303=39, srv630319677=80, srv1733863686=35, srv1790963242=42, srv818002073=96, srv175219432=40, srv1286264611=19, srv1052367495=2, srv2083247464=57, srv1947600258=47, srv807300751=94, srv1131750004=11, srv2039509502=52, srv1598975481=31, srv362015878=69, srv597781971=79, srv1951131263=48, srv1945272266=46, srv1314302088=21, srv309543465=66, srv1135165098=12, srv436430590=73, srv677302791=84, srv931546925=98, srv713080285=86, srv1202711863=15, srv226885252=62, srv418979864=70, srv798014095=93, srv941450114=99, srv179687780=43, srv218247140=61, srv1489557885=29, srv1297922342=20, srv816043509=95, srv1479042501=28, srv1272286185=18, srv1389037701=24, srv739382439=88, srv110135646=8, srv2116201054=59, srv30859695=65, srv53021258=78, srv304863471=64, srv1641576971=32, srv1092234546=6, srv1172359170=13, srv1127644970=10, srv1118227721=9, srv2093195589=58, srv1759233737=41, srv1823856869=44, srv268018004=63, srv1866462693=45, srv76778478=90, srv2141118044=60, srv1094708566=7, srv1062801435=3, srv461979205=75, srv424720750=72, srv440304999=74, srv2065333489=54, srv779286202=92, srv881811220=97, srv696448729=85, srv1224987328=16, srv1469399910=27, srv1033476424=1, srv730861495=87, srv357487835=68, srv650005157=82, srv1253375993=17, srv644037916=81, srv1683108487=33, srv493962020=77, srv1731159213=34, srv651464475=83, srv205818251=53, srv1083478858=5, srv477534705=76, srv1012633449=0, srv1465228820=26, srv323133853=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-07 13:53:16,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-07 13:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-07 13:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-07 13:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-07 13:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-07 13:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-07 13:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-07 13:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-07 13:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-07 13:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-07 13:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-07 13:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-07 13:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-07 13:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-07 13:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-07 13:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-07 13:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-07 13:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-07 13:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-07 13:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-07 13:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-07 13:53:16,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-07 13:53:16,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-07 13:53:16,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-07 13:53:16,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-07 13:53:16,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-07 13:53:16,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-07 13:53:16,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-07 13:53:16,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-07 13:53:16,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-07 13:53:16,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-07 13:53:16,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-07 13:53:16,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-07 13:53:16,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-07 13:53:16,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-07 13:53:16,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-07 13:53:16,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-07 13:53:16,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-07 13:53:16,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-07 13:53:16,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-07 13:53:16,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-07 13:53:16,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-07 13:53:16,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-07 13:53:16,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-07 13:53:16,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-07 13:53:16,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-07 13:53:16,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-07 13:53:16,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-07 13:53:16,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-07 13:53:16,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-07 13:53:16,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-07 13:53:16,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-07 13:53:16,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-07 13:53:16,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-07 13:53:16,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-07 13:53:16,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-07 13:53:16,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-07 13:53:16,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-07 13:53:16,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-07 13:53:16,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-07 13:53:16,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-07 13:53:16,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-07 13:53:16,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-07 13:53:16,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-07 13:53:16,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-07 13:53:16,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-07 13:53:16,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-07 13:53:16,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-07 13:53:16,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-07 13:53:16,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-07 13:53:16,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-07 13:53:16,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-07 13:53:16,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-07 13:53:16,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-07 13:53:16,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-07 13:53:16,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-07 13:53:16,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-07 13:53:16,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-07 13:53:16,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-07 13:53:16,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-07 13:53:16,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-07 13:53:16,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-07 13:53:16,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-07 13:53:16,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-07 13:53:16,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-07 13:53:16,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-07 13:53:16,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-07 13:53:16,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-07 13:53:16,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-07 13:53:16,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-07 13:53:16,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-07 13:53:16,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-07 13:53:16,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-07 13:53:16,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-07 13:53:16,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-07 13:53:16,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-07 13:53:16,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-07 13:53:16,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-07 13:53:16,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-07 13:53:16,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-07 13:53:16,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-07 13:53:16,384 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03928561468447832, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054154038932104895, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03928561468447832, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17433333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24916666666666668, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:53:16,385 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-07 13:53:16,385 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17981305601809783, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03928561468447832, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054154038932104895, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03928561468447832, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17433333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24916666666666668, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-07 13:53:16,710 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 431 ms to try 5000 different iterations. Found a solution that moves 2708 regions; Going from a computed imbalance of 0.17981305601809783 to a new imbalance of 0.016342096755518794. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02230623848491836); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030715376335548022, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.3008888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02230623848491836); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.049416666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:53:16,723 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv76778478:74 , srv1378746570:78 , srv2116201054:78 , srv493962020:79 , srv477534705:80 , srv597781971:80 , srv1127644970:81 , srv309543465:81 , srv424720750:81 , srv1195608128:82 , srv424113286:82 , srv650005157:82 , srv730861495:82 , srv1033476424:83 , srv1083478858:83 , srv110135646:83 , srv1598975481:83 , srv1735940010:83 , srv630319677:83 , srv651464475:83 , srv677302791:83 , srv739382439:83 , srv1253375993:84 , srv1389037701:84 , srv1469399910:84 , srv1731159213:84 , srv1823856869:84 , srv2013680364:84 , srv30859695:84 , srv418979864:84 , srv779286202:84 , srv798014095:84 , srv816043509:84 , srv1172359170:85 , srv1272286185:85 , srv1733863686:85 , srv175219432:85 , srv179687780:85 , srv205818251:85 , srv2065333489:85 , srv2093195589:85 , srv53021258:85 , srv696448729:85 , srv1118227721:86 , srv1131750004:86 , srv1224987328:86 , srv1286264611:86 , srv1297922342:86 , srv1465228820:86 , srv1479042501:86 , srv1683108487:86 , srv1736213305:86 , srv2068728778:86 , srv218247140:86 , srv440304999:86 , srv807300751:86 , srv1012633449:87 , srv1092234546:87 , srv1094708566:87 , srv1749364058:87 , srv1759233737:87 , srv1866462693:87 , srv2070650608:87 , srv2141118044:87 , srv226885252:87 , srv304863471:87 , srv461979205:87 , srv644037916:87 , srv713080285:87 , srv744878033:87 , srv77122411:87 , srv1135165098:88 , srv1314302088:88 , srv1428177451:88 , srv1489557885:88 , srv1594293913:88 , srv1945272266:88 , srv1947600258:88 , srv1951131263:88 , srv2014157401:88 , srv2030957120:88 , srv2039509502:88 , srv357487835:88 , srv362015878:88 , srv818002073:88 , srv931546925:88 , srv1062801435:89 , srv1202711863:89 , srv1321358964:89 , srv1641576971:89 , srv175022303:89 , srv1790963242:89 , srv2083247464:89 , srv268018004:89 , srv881811220:89 , srv1052367495:90 , srv436430590:90 , srv323133853:91 , srv941450114:91 , srv1081059137:533 } 2023-08-07 13:53:16,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-07 13:53:16,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1736213305=37, srv2013680364=49, srv1378746570=23, srv424113286=71, srv1321358964=22, srv1428177451=25, srv2068728778=55, srv77122411=91, srv744878033=89, srv2014157401=50, srv1594293913=30, srv1081059137=4, srv1195608128=14, srv2030957120=51, srv1749364058=38, srv2070650608=56, srv1735940010=36, srv175022303=39, srv630319677=80, srv1733863686=35, srv1790963242=42, srv818002073=96, srv175219432=40, srv1286264611=19, srv1052367495=2, srv2083247464=57, srv1947600258=47, srv807300751=94, srv1131750004=11, srv2039509502=52, srv1598975481=31, srv362015878=69, srv597781971=79, srv1951131263=48, srv1945272266=46, srv1314302088=21, srv309543465=66, srv1135165098=12, srv436430590=73, srv677302791=84, srv931546925=98, srv713080285=86, srv1202711863=15, srv226885252=62, srv418979864=70, srv798014095=93, srv941450114=99, srv179687780=43, srv218247140=61, srv1489557885=29, srv1297922342=20, srv816043509=95, srv1479042501=28, srv1272286185=18, srv1389037701=24, srv739382439=88, srv110135646=8, srv2116201054=59, srv30859695=65, srv53021258=78, srv304863471=64, srv1641576971=32, srv1092234546=6, srv1172359170=13, srv1127644970=10, srv1118227721=9, srv2093195589=58, srv1759233737=41, srv1823856869=44, srv268018004=63, srv1866462693=45, srv76778478=90, srv2141118044=60, srv1094708566=7, srv1062801435=3, srv461979205=75, srv424720750=72, srv440304999=74, srv2065333489=54, srv779286202=92, srv881811220=97, srv696448729=85, srv1224987328=16, srv1469399910=27, srv1033476424=1, srv730861495=87, srv357487835=68, srv650005157=82, srv1253375993=17, srv644037916=81, srv1683108487=33, srv493962020=77, srv1731159213=34, srv651464475=83, srv205818251=53, srv1083478858=5, srv477534705=76, srv1012633449=0, srv1465228820=26, srv323133853=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-07 13:53:16,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-07 13:53:16,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-07 13:53:16,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-07 13:53:16,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-07 13:53:16,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-07 13:53:16,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-07 13:53:16,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-07 13:53:16,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-07 13:53:16,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-07 13:53:16,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-07 13:53:16,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-07 13:53:16,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-07 13:53:16,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-07 13:53:16,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-07 13:53:16,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-07 13:53:16,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-07 13:53:16,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-07 13:53:16,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-07 13:53:16,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-07 13:53:16,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-07 13:53:16,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-07 13:53:16,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-07 13:53:16,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-07 13:53:16,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-07 13:53:16,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-07 13:53:16,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-07 13:53:16,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-07 13:53:16,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-07 13:53:16,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-07 13:53:16,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-07 13:53:16,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-07 13:53:16,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-07 13:53:16,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-07 13:53:16,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-07 13:53:16,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-07 13:53:16,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-07 13:53:16,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-07 13:53:16,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-07 13:53:16,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-07 13:53:16,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-07 13:53:16,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-07 13:53:16,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-07 13:53:16,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-07 13:53:16,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-07 13:53:16,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-07 13:53:16,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-07 13:53:16,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-07 13:53:16,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-07 13:53:16,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-07 13:53:16,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-07 13:53:16,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-07 13:53:16,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-07 13:53:16,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-07 13:53:16,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-07 13:53:16,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-07 13:53:16,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-07 13:53:16,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-07 13:53:16,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-07 13:53:16,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-07 13:53:16,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-07 13:53:16,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-07 13:53:16,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-07 13:53:16,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-07 13:53:16,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-07 13:53:16,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-07 13:53:16,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-07 13:53:16,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-07 13:53:16,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-07 13:53:16,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-07 13:53:16,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-07 13:53:16,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-07 13:53:16,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-07 13:53:16,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-07 13:53:16,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-07 13:53:16,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-07 13:53:16,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-07 13:53:16,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-07 13:53:16,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-07 13:53:16,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-07 13:53:16,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-07 13:53:16,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-07 13:53:16,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-07 13:53:16,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-07 13:53:16,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-07 13:53:16,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-07 13:53:16,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-07 13:53:16,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-07 13:53:16,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-07 13:53:16,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-07 13:53:16,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-07 13:53:16,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-07 13:53:16,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-07 13:53:16,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-07 13:53:16,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-07 13:53:16,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-07 13:53:16,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-07 13:53:16,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-07 13:53:16,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-07 13:53:16,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-07 13:53:16,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-07 13:53:16,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-07 13:53:16,912 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02230623848491836); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030715376335548022, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02230623848491836); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.049416666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:53:16,913 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-07 13:53:16,913 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01632313493423966, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02230623848491836); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030715376335548022, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02230623848491836); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.049416666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-07 13:53:17,239 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 457 ms to try 5000 different iterations. Found a solution that moves 1081 regions; Going from a computed imbalance of 0.01632313493423966 to a new imbalance of 7.653079528947358E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005665577237325316); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005605833101216811); MoveCostFunction : (multiplier=7.0, imbalance=0.12011111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005665577237325316); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:53:17,241 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv323133853:79 , srv1641576971:81 , srv1790963242:81 , srv2030957120:81 , srv268018004:82 , srv1092234546:83 , srv205818251:83 , srv2068728778:83 , srv2083247464:83 , srv304863471:83 , srv881811220:83 , srv941450114:83 , srv1131750004:84 , srv1272286185:84 , srv1489557885:84 , srv175219432:84 , srv1947600258:84 , srv2093195589:84 , srv226885252:84 , srv309543465:84 , srv713080285:84 , srv76778478:84 , srv77122411:84 , srv1062801435:85 , srv1094708566:85 , srv110135646:85 , srv1135165098:85 , srv1172359170:85 , srv1202711863:85 , srv1465228820:85 , srv1469399910:85 , srv1731159213:85 , srv1749364058:85 , srv179687780:85 , srv1951131263:85 , srv2070650608:85 , srv357487835:85 , srv436430590:85 , srv493962020:85 , srv53021258:85 , srv597781971:85 , srv644037916:85 , srv779286202:85 , srv798014095:85 , srv818002073:85 , srv1012633449:86 , srv1033476424:86 , srv1052367495:86 , srv1118227721:86 , srv1224987328:86 , srv1297922342:86 , srv1321358964:86 , srv1428177451:86 , srv1479042501:86 , srv1594293913:86 , srv1735940010:86 , srv1736213305:86 , srv175022303:86 , srv1823856869:86 , srv1866462693:86 , srv2014157401:86 , srv2039509502:86 , srv2065333489:86 , srv362015878:86 , srv424113286:86 , srv424720750:86 , srv440304999:86 , srv461979205:86 , srv807300751:86 , srv1083478858:87 , srv1195608128:87 , srv1253375993:87 , srv1314302088:87 , srv1378746570:87 , srv1389037701:87 , srv1683108487:87 , srv1759233737:87 , srv1945272266:87 , srv2116201054:87 , srv2141118044:87 , srv218247140:87 , srv30859695:87 , srv477534705:87 , srv630319677:87 , srv651464475:87 , srv677302791:87 , srv696448729:87 , srv739382439:87 , srv744878033:87 , srv1127644970:88 , srv1286264611:88 , srv1598975481:88 , srv1733863686:88 , srv2013680364:88 , srv418979864:88 , srv650005157:88 , srv816043509:88 , srv931546925:88 , srv730861495:89 , srv1081059137:536 } 2023-08-07 13:53:17,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-07 13:53:17,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1736213305=37, srv2013680364=49, srv1378746570=23, srv424113286=71, srv1321358964=22, srv1428177451=25, srv2068728778=55, srv77122411=91, srv744878033=89, srv2014157401=50, srv1594293913=30, srv1081059137=4, srv1195608128=14, srv2030957120=51, srv1749364058=38, srv2070650608=56, srv1735940010=36, srv175022303=39, srv630319677=80, srv1733863686=35, srv1790963242=42, srv818002073=96, srv175219432=40, srv1286264611=19, srv1052367495=2, srv2083247464=57, srv1947600258=47, srv807300751=94, srv1131750004=11, srv2039509502=52, srv1598975481=31, srv362015878=69, srv597781971=79, srv1951131263=48, srv1945272266=46, srv1314302088=21, srv309543465=66, srv1135165098=12, srv436430590=73, srv677302791=84, srv931546925=98, srv713080285=86, srv1202711863=15, srv226885252=62, srv418979864=70, srv798014095=93, srv941450114=99, srv179687780=43, srv218247140=61, srv1489557885=29, srv1297922342=20, srv816043509=95, srv1479042501=28, srv1272286185=18, srv1389037701=24, srv739382439=88, srv110135646=8, srv2116201054=59, srv30859695=65, srv53021258=78, srv304863471=64, srv1641576971=32, srv1092234546=6, srv1172359170=13, srv1127644970=10, srv1118227721=9, srv2093195589=58, srv1759233737=41, srv1823856869=44, srv268018004=63, srv1866462693=45, srv76778478=90, srv2141118044=60, srv1094708566=7, srv1062801435=3, srv461979205=75, srv424720750=72, srv440304999=74, srv2065333489=54, srv779286202=92, srv881811220=97, srv696448729=85, srv1224987328=16, srv1469399910=27, srv1033476424=1, srv730861495=87, srv357487835=68, srv650005157=82, srv1253375993=17, srv644037916=81, srv1683108487=33, srv493962020=77, srv1731159213=34, srv651464475=83, srv205818251=53, srv1083478858=5, srv477534705=76, srv1012633449=0, srv1465228820=26, srv323133853=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-07 13:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-07 13:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-07 13:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-07 13:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-07 13:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-07 13:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-07 13:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-07 13:53:17,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-07 13:53:17,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-07 13:53:17,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-07 13:53:17,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-07 13:53:17,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-07 13:53:17,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-07 13:53:17,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-07 13:53:17,387 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005665577237325316); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005605833101216811); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005665577237325316); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:53:17,387 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-07 13:53:17,388 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.577386291123345E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005665577237325316); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005605833101216811); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005665577237325316); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-07 13:53:17,760 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 472 ms to try 5000 different iterations. Found a solution that moves 248 regions; Going from a computed imbalance of 7.577386291123345E-4 to a new imbalance of 4.1840552373555654E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.027555555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:53:17,761 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1202711863:83 , srv1428177451:83 , srv1683108487:83 , srv362015878:83 , srv1033476424:84 , srv1052367495:84 , srv1062801435:84 , srv1092234546:84 , srv1118227721:84 , srv1127644970:84 , srv1135165098:84 , srv1253375993:84 , srv1286264611:84 , srv1314302088:84 , srv1378746570:84 , srv1389037701:84 , srv1465228820:84 , srv1479042501:84 , srv1598975481:84 , srv1731159213:84 , srv1736213305:84 , srv175022303:84 , srv175219432:84 , srv179687780:84 , srv1823856869:84 , srv1947600258:84 , srv2039509502:84 , srv2068728778:84 , srv2083247464:84 , srv2141118044:84 , srv218247140:84 , srv309543465:84 , srv323133853:84 , srv357487835:84 , srv418979864:84 , srv424113286:84 , srv436430590:84 , srv461979205:84 , srv477534705:84 , srv493962020:84 , srv53021258:84 , srv630319677:84 , srv650005157:84 , srv651464475:84 , srv677302791:84 , srv696448729:84 , srv730861495:84 , srv744878033:84 , srv816043509:84 , srv881811220:84 , srv931546925:84 , srv941450114:84 , srv1012633449:85 , srv1094708566:85 , srv110135646:85 , srv1131750004:85 , srv1172359170:85 , srv1195608128:85 , srv1224987328:85 , srv1272286185:85 , srv1297922342:85 , srv1321358964:85 , srv1469399910:85 , srv1594293913:85 , srv1641576971:85 , srv1733863686:85 , srv1735940010:85 , srv1749364058:85 , srv1759233737:85 , srv1790963242:85 , srv1866462693:85 , srv1945272266:85 , srv1951131263:85 , srv2013680364:85 , srv2014157401:85 , srv205818251:85 , srv2065333489:85 , srv2070650608:85 , srv2093195589:85 , srv2116201054:85 , srv226885252:85 , srv268018004:85 , srv304863471:85 , srv30859695:85 , srv440304999:85 , srv597781971:85 , srv644037916:85 , srv713080285:85 , srv739382439:85 , srv76778478:85 , srv77122411:85 , srv779286202:85 , srv807300751:85 , srv1083478858:86 , srv2030957120:86 , srv424720750:86 , srv798014095:86 , srv818002073:86 , srv1489557885:87 , srv1081059137:634 } 2023-08-07 13:53:17,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-07 13:53:17,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1736213305=37, srv2013680364=49, srv1378746570=23, srv424113286=71, srv1321358964=22, srv1428177451=25, srv2068728778=55, srv77122411=91, srv744878033=89, srv2014157401=50, srv1594293913=30, srv1081059137=4, srv1195608128=14, srv2030957120=51, srv1749364058=38, srv2070650608=56, srv1735940010=36, srv175022303=39, srv630319677=80, srv1733863686=35, srv1790963242=42, srv818002073=96, srv175219432=40, srv1286264611=19, srv1052367495=2, srv2083247464=57, srv1947600258=47, srv807300751=94, srv1131750004=11, srv2039509502=52, srv1598975481=31, srv362015878=69, srv597781971=79, srv1951131263=48, srv1945272266=46, srv1314302088=21, srv309543465=66, srv1135165098=12, srv436430590=73, srv677302791=84, srv931546925=98, srv713080285=86, srv1202711863=15, srv226885252=62, srv418979864=70, srv798014095=93, srv941450114=99, srv179687780=43, srv218247140=61, srv1489557885=29, srv1297922342=20, srv816043509=95, srv1479042501=28, srv1272286185=18, srv1389037701=24, srv739382439=88, srv110135646=8, srv2116201054=59, srv30859695=65, srv53021258=78, srv304863471=64, srv1641576971=32, srv1092234546=6, srv1172359170=13, srv1127644970=10, srv1118227721=9, srv2093195589=58, srv1759233737=41, srv1823856869=44, srv268018004=63, srv1866462693=45, srv76778478=90, srv2141118044=60, srv1094708566=7, srv1062801435=3, srv461979205=75, srv424720750=72, srv440304999=74, srv2065333489=54, srv779286202=92, srv881811220=97, srv696448729=85, srv1224987328=16, srv1469399910=27, srv1033476424=1, srv730861495=87, srv357487835=68, srv650005157=82, srv1253375993=17, srv644037916=81, srv1683108487=33, srv493962020=77, srv1731159213=34, srv651464475=83, srv205818251=53, srv1083478858=5, srv477534705=76, srv1012633449=0, srv1465228820=26, srv323133853=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-07 13:53:17,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-07 13:53:17,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-07 13:53:17,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-07 13:53:17,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-07 13:53:17,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-07 13:53:17,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-07 13:53:17,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-07 13:53:17,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-07 13:53:17,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-07 13:53:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-07 13:53:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-07 13:53:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-07 13:53:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-07 13:53:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-07 13:53:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-07 13:53:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-07 13:53:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-07 13:53:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-07 13:53:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-07 13:53:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-07 13:53:17,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-07 13:53:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-07 13:53:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-07 13:53:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-07 13:53:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-07 13:53:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-07 13:53:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-07 13:53:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-07 13:53:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-07 13:53:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-07 13:53:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-07 13:53:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-07 13:53:17,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-07 13:53:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-07 13:53:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-07 13:53:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-07 13:53:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-07 13:53:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-07 13:53:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-07 13:53:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-07 13:53:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-07 13:53:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-07 13:53:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-07 13:53:17,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-07 13:53:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-07 13:53:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-07 13:53:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-07 13:53:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-07 13:53:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-07 13:53:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-07 13:53:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-07 13:53:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-07 13:53:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-07 13:53:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-07 13:53:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-07 13:53:17,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-07 13:53:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-07 13:53:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-07 13:53:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-07 13:53:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-07 13:53:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-07 13:53:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-07 13:53:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-07 13:53:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-07 13:53:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-07 13:53:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-07 13:53:17,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-07 13:53:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-07 13:53:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-07 13:53:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-07 13:53:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-07 13:53:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-07 13:53:17,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-07 13:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-07 13:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-07 13:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-07 13:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-07 13:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-07 13:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-07 13:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-07 13:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-07 13:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-07 13:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-07 13:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-07 13:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-07 13:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-07 13:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-07 13:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-07 13:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-07 13:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-07 13:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-07 13:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-07 13:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-07 13:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-07 13:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-07 13:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-07 13:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-07 13:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-07 13:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-07 13:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-07 13:53:17,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-07 13:53:17,931 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:53:17,932 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-07 13:53:17,932 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.0104019257889135E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-07 13:53:18,217 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 412 ms to try 5000 different iterations. Found a solution that moves 24 regions; Going from a computed imbalance of 4.0104019257889135E-5 to a new imbalance of 6.912120645670642E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0026666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:53:18,217 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1062801435:83 , srv1083478858:83 , srv1286264611:83 , srv1321358964:83 , srv1489557885:83 , srv1790963242:83 , srv1945272266:83 , srv2013680364:83 , srv268018004:83 , srv650005157:83 , srv1012633449:84 , srv1033476424:84 , srv1052367495:84 , srv1092234546:84 , srv1094708566:84 , srv110135646:84 , srv1118227721:84 , srv1127644970:84 , srv1131750004:84 , srv1172359170:84 , srv1195608128:84 , srv1202711863:84 , srv1224987328:84 , srv1253375993:84 , srv1272286185:84 , srv1297922342:84 , srv1314302088:84 , srv1378746570:84 , srv1428177451:84 , srv1469399910:84 , srv1594293913:84 , srv1598975481:84 , srv1683108487:84 , srv1731159213:84 , srv1733863686:84 , srv1735940010:84 , srv1749364058:84 , srv175022303:84 , srv1759233737:84 , srv179687780:84 , srv1823856869:84 , srv1866462693:84 , srv1947600258:84 , srv1951131263:84 , srv2014157401:84 , srv2030957120:84 , srv2039509502:84 , srv205818251:84 , srv2065333489:84 , srv2068728778:84 , srv2070650608:84 , srv2083247464:84 , srv2093195589:84 , srv2116201054:84 , srv2141118044:84 , srv218247140:84 , srv226885252:84 , srv304863471:84 , srv30859695:84 , srv309543465:84 , srv323133853:84 , srv357487835:84 , srv418979864:84 , srv424113286:84 , srv424720750:84 , srv436430590:84 , srv440304999:84 , srv461979205:84 , srv493962020:84 , srv53021258:84 , srv597781971:84 , srv630319677:84 , srv644037916:84 , srv651464475:84 , srv696448729:84 , srv713080285:84 , srv730861495:84 , srv739382439:84 , srv744878033:84 , srv76778478:84 , srv77122411:84 , srv779286202:84 , srv798014095:84 , srv807300751:84 , srv816043509:84 , srv818002073:84 , srv881811220:84 , srv931546925:84 , srv941450114:84 , srv1135165098:85 , srv1389037701:85 , srv1465228820:85 , srv1479042501:85 , srv1641576971:85 , srv1736213305:85 , srv175219432:85 , srv362015878:85 , srv477534705:85 , srv677302791:85 , srv1081059137:684 } 2023-08-07 13:53:18,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-07 13:53:18,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1736213305=37, srv2013680364=49, srv1378746570=23, srv424113286=71, srv1321358964=22, srv1428177451=25, srv2068728778=55, srv77122411=91, srv744878033=89, srv2014157401=50, srv1594293913=30, srv1081059137=4, srv1195608128=14, srv2030957120=51, srv1749364058=38, srv2070650608=56, srv1735940010=36, srv175022303=39, srv630319677=80, srv1733863686=35, srv1790963242=42, srv818002073=96, srv175219432=40, srv1286264611=19, srv1052367495=2, srv2083247464=57, srv1947600258=47, srv807300751=94, srv1131750004=11, srv2039509502=52, srv1598975481=31, srv362015878=69, srv597781971=79, srv1951131263=48, srv1945272266=46, srv1314302088=21, srv309543465=66, srv1135165098=12, srv436430590=73, srv677302791=84, srv931546925=98, srv713080285=86, srv1202711863=15, srv226885252=62, srv418979864=70, srv798014095=93, srv941450114=99, srv179687780=43, srv218247140=61, srv1489557885=29, srv1297922342=20, srv816043509=95, srv1479042501=28, srv1272286185=18, srv1389037701=24, srv739382439=88, srv110135646=8, srv2116201054=59, srv30859695=65, srv53021258=78, srv304863471=64, srv1641576971=32, srv1092234546=6, srv1172359170=13, srv1127644970=10, srv1118227721=9, srv2093195589=58, srv1759233737=41, srv1823856869=44, srv268018004=63, srv1866462693=45, srv76778478=90, srv2141118044=60, srv1094708566=7, srv1062801435=3, srv461979205=75, srv424720750=72, srv440304999=74, srv2065333489=54, srv779286202=92, srv881811220=97, srv696448729=85, srv1224987328=16, srv1469399910=27, srv1033476424=1, srv730861495=87, srv357487835=68, srv650005157=82, srv1253375993=17, srv644037916=81, srv1683108487=33, srv493962020=77, srv1731159213=34, srv651464475=83, srv205818251=53, srv1083478858=5, srv477534705=76, srv1012633449=0, srv1465228820=26, srv323133853=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-07 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-07 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-07 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-07 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-07 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-07 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-07 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-07 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-07 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-07 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-07 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-07 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-07 13:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-07 13:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-07 13:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-07 13:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-07 13:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-07 13:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-07 13:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-07 13:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-07 13:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-07 13:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-07 13:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-07 13:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-07 13:53:18,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-07 13:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-07 13:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-07 13:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-07 13:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-07 13:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-07 13:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-07 13:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-07 13:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-07 13:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-07 13:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-07 13:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-07 13:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-07 13:53:18,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-07 13:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-07 13:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-07 13:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-07 13:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-07 13:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-07 13:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-07 13:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-07 13:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-07 13:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-07 13:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-07 13:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-07 13:53:18,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-07 13:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-07 13:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-07 13:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-07 13:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-07 13:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-07 13:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-07 13:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-07 13:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-07 13:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-07 13:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-07 13:53:18,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-07 13:53:18,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-07 13:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-07 13:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-07 13:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-07 13:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-07 13:53:18,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-07 13:53:18,377 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:53:18,378 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1062801435:83 , srv1083478858:83 , srv1286264611:83 , srv1321358964:83 , srv1489557885:83 , srv1790963242:83 , srv1945272266:83 , srv2013680364:83 , srv268018004:83 , srv650005157:83 , srv1012633449:84 , srv1033476424:84 , srv1052367495:84 , srv1092234546:84 , srv1094708566:84 , srv110135646:84 , srv1118227721:84 , srv1127644970:84 , srv1131750004:84 , srv1172359170:84 , srv1195608128:84 , srv1202711863:84 , srv1224987328:84 , srv1253375993:84 , srv1272286185:84 , srv1297922342:84 , srv1314302088:84 , srv1378746570:84 , srv1428177451:84 , srv1469399910:84 , srv1594293913:84 , srv1598975481:84 , srv1683108487:84 , srv1731159213:84 , srv1733863686:84 , srv1735940010:84 , srv1749364058:84 , srv175022303:84 , srv1759233737:84 , srv179687780:84 , srv1823856869:84 , srv1866462693:84 , srv1947600258:84 , srv1951131263:84 , srv2014157401:84 , srv2030957120:84 , srv2039509502:84 , srv205818251:84 , srv2065333489:84 , srv2068728778:84 , srv2070650608:84 , srv2083247464:84 , srv2093195589:84 , srv2116201054:84 , srv2141118044:84 , srv218247140:84 , srv226885252:84 , srv304863471:84 , srv30859695:84 , srv309543465:84 , srv323133853:84 , srv357487835:84 , srv418979864:84 , srv424113286:84 , srv424720750:84 , srv436430590:84 , srv440304999:84 , srv461979205:84 , srv493962020:84 , srv53021258:84 , srv597781971:84 , srv630319677:84 , srv644037916:84 , srv651464475:84 , srv696448729:84 , srv713080285:84 , srv730861495:84 , srv739382439:84 , srv744878033:84 , srv76778478:84 , srv77122411:84 , srv779286202:84 , srv798014095:84 , srv807300751:84 , srv816043509:84 , srv818002073:84 , srv881811220:84 , srv931546925:84 , srv941450114:84 , srv1135165098:85 , srv1389037701:85 , srv1465228820:85 , srv1479042501:85 , srv1641576971:85 , srv1736213305:85 , srv175219432:85 , srv362015878:85 , srv477534705:85 , srv677302791:85 , srv1081059137:684 } 2023-08-07 13:53:18,409 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=302 (was 284) - SystemLoadAverage LEAK? -, ProcessCount=174 (was 174), AvailableMemoryMB=2563 (was 3406) 2023-08-07 13:53:18,417 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=302, ProcessCount=174, AvailableMemoryMB=2562 2023-08-07 13:53:18,418 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-07 13:53:18,418 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-07 13:53:18,427 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1044881683:84 , srv1086302527:84 , srv1350946818:84 , srv1579086361:84 , srv1620348165:84 , srv1624942472:84 , srv1786482687:84 , srv1813941420:84 , srv1878446204:84 , srv1932211969:84 , srv1965399372:84 , srv1992251255:84 , srv2016123985:84 , srv2035461546:84 , srv2118249601:84 , srv247977765:84 , srv287065202:84 , srv402892859:84 , srv444537399:84 , srv521777749:84 , srv541059752:84 , srv661195934:84 , srv667443431:84 , srv710409105:84 , srv817196713:84 , srv858833798:84 , srv869296027:84 , srv930330551:84 , srv973244160:84 , srv1039678876:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-07 13:53:18,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-07 13:53:18,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1624942472=6, srv667443431=23, srv521777749=20, srv402892859=18, srv2035461546=14, srv661195934=22, srv1992251255=12, srv869296027=27, srv1086302527=2, srv1813941420=8, srv1965399372=11, srv1878446204=9, srv710409105=24, srv858833798=26, srv1579086361=4, srv1350946818=3, srv2118249601=15, srv247977765=16, srv930330551=28, srv287065202=17, srv2016123985=13, srv817196713=25, srv1044881683=1, srv1932211969=10, srv973244160=29, srv1039678876=0, srv1620348165=5, srv444537399=19, srv1786482687=7, srv541059752=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-07 13:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-07 13:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-07 13:53:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-07 13:53:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-07 13:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-07 13:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-07 13:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-07 13:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-07 13:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-07 13:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-07 13:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-07 13:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-07 13:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-07 13:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-07 13:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-07 13:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-07 13:53:18,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-07 13:53:18,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-07 13:53:18,462 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.08792313923677, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:53:18,462 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991247008620506, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.08792313923677, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-07 13:55:18,447 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6098901 different iterations. Found a solution that moves 1848 regions; Going from a computed imbalance of 0.991247008620506 to a new imbalance of 4.313324190526491E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:55:18,455 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1039678876:90 , srv1044881683:90 , srv1086302527:90 , srv1350946818:90 , srv1579086361:90 , srv1620348165:90 , srv1624942472:90 , srv1786482687:90 , srv1813941420:90 , srv1878446204:90 , srv1932211969:90 , srv1965399372:90 , srv1992251255:90 , srv2016123985:90 , srv2035461546:90 , srv2118249601:90 , srv247977765:90 , srv287065202:90 , srv402892859:90 , srv444537399:90 , srv521777749:90 , srv541059752:90 , srv661195934:90 , srv667443431:90 , srv710409105:90 , srv817196713:90 , srv858833798:90 , srv869296027:90 , srv930330551:90 , srv973244160:90 } 2023-08-07 13:55:18,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-07 13:55:18,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1624942472=6, srv667443431=23, srv521777749=20, srv402892859=18, srv2035461546=14, srv661195934=22, srv1992251255=12, srv869296027=27, srv1086302527=2, srv1813941420=8, srv1965399372=11, srv1878446204=9, srv710409105=24, srv858833798=26, srv1579086361=4, srv1350946818=3, srv2118249601=15, srv247977765=16, srv930330551=28, srv287065202=17, srv2016123985=13, srv817196713=25, srv1044881683=1, srv1932211969=10, srv973244160=29, srv1039678876=0, srv1620348165=5, srv444537399=19, srv1786482687=7, srv541059752=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-07 13:55:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-07 13:55:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-07 13:55:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-07 13:55:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-07 13:55:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-07 13:55:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-07 13:55:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-07 13:55:18,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-07 13:55:18,481 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-07 13:55:18,482 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1039678876:90 , srv1044881683:90 , srv1086302527:90 , srv1350946818:90 , srv1579086361:90 , srv1620348165:90 , srv1624942472:90 , srv1786482687:90 , srv1813941420:90 , srv1878446204:90 , srv1932211969:90 , srv1965399372:90 , srv1992251255:90 , srv2016123985:90 , srv2035461546:90 , srv2118249601:90 , srv247977765:90 , srv287065202:90 , srv402892859:90 , srv444537399:90 , srv521777749:90 , srv541059752:90 , srv661195934:90 , srv667443431:90 , srv710409105:90 , srv817196713:90 , srv858833798:90 , srv869296027:90 , srv930330551:90 , srv973244160:90 } 2023-08-07 13:55:18,501 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 302), ProcessCount=171 (was 174), AvailableMemoryMB=2787 (was 2562) - AvailableMemoryMB LEAK? -