2023-04-08 11:56:37,308 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-04-08 11:56:37,622 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-08 11:56:37,741 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-08 11:56:37,806 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-08 11:56:37,813 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-08 11:56:37,825 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=421, ProcessCount=169, AvailableMemoryMB=2515 2023-04-08 11:56:37,827 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-08 11:56:37,828 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-08 11:56:37,947 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1014405857:84 , srv1081103086:84 , srv1125802450:84 , srv1164647731:84 , srv1183153101:84 , srv118649404:84 , srv1196861416:84 , srv1212258769:84 , srv1218863306:84 , srv1219788952:84 , srv1243162198:84 , srv1266333902:84 , srv1276893234:84 , srv1310263082:84 , srv1312275765:84 , srv1391304479:84 , srv1394332344:84 , srv1403179094:84 , srv1413864525:84 , srv1419575123:84 , srv1450207905:84 , srv1466946618:84 , srv1473581608:84 , srv1475080100:84 , srv1494078914:84 , srv1497193115:84 , srv1513178406:84 , srv155340045:84 , srv1557540017:84 , srv1571058221:84 , srv157926863:84 , srv1581055140:84 , srv1583712728:84 , srv1583914036:84 , srv159089925:84 , srv1616286964:84 , srv1664045205:84 , srv1685055574:84 , srv1707073217:84 , srv1735035005:84 , srv1741326866:84 , srv174374296:84 , srv1751846227:84 , srv1780150796:84 , srv180797568:84 , srv1810352926:84 , srv1851841094:84 , srv1904948188:84 , srv1916872682:84 , srv1951117460:84 , srv1984065800:84 , srv2024828962:84 , srv2081605848:84 , srv2095545956:84 , srv2101810908:84 , srv2103717442:84 , srv2119807533:84 , srv2134096162:84 , srv2134563337:84 , srv22188:84 , srv230275016:84 , srv279633160:84 , srv331235399:84 , srv351682379:84 , srv352141811:84 , srv358530253:84 , srv367332841:84 , srv368952153:84 , srv370658453:84 , srv383029460:84 , srv38405265:84 , srv392485198:84 , srv422647470:84 , srv443386290:84 , srv49020945:84 , srv490868940:84 , srv499487159:84 , srv503674964:84 , srv516333159:84 , srv523539564:84 , srv599658034:84 , srv608406283:84 , srv61277761:84 , srv639063953:84 , srv666792107:84 , srv685559277:84 , srv69147545:84 , srv692064328:84 , srv736725489:84 , srv737549097:84 , srv756131251:84 , srv757499536:84 , srv765023097:84 , srv770641298:84 , srv820456167:84 , srv825144016:84 , srv858618316:84 , srv920108541:84 , srv94348159:84 , srv889366805:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-04-08 11:56:37,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-08 11:56:37,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1664045205=36, srv1571058221=29, srv180797568=44, srv2134563337=58, srv499487159=76, srv1219788952=9, srv2081605848=52, srv2024828962=51, srv157926863=30, srv503674964=77, srv443386290=73, srv1685055574=37, srv516333159=78, srv230275016=60, srv1310263082=13, srv2119807533=56, srv61277761=82, srv736725489=88, srv1741326866=40, srv770641298=93, srv118649404=5, srv2095545956=53, srv1466946618=21, srv756131251=90, srv1419575123=19, srv1218863306=8, srv1212258769=7, srv358530253=65, srv1904948188=47, srv666792107=84, srv174374296=41, srv1183153101=4, srv1583712728=32, srv1450207905=20, srv352141811=64, srv1473581608=22, srv1513178406=26, srv2101810908=54, srv1164647731=3, srv490868940=75, srv858618316=96, srv351682379=63, srv692064328=87, srv1916872682=48, srv920108541=98, srv757499536=91, srv639063953=83, srv1413864525=18, srv392485198=71, srv1403179094=17, srv889366805=97, srv38405265=70, srv1014405857=0, srv608406283=81, srv1707073217=38, srv1266333902=11, srv69147545=86, srv1751846227=42, srv331235399=62, srv1951117460=49, srv49020945=74, srv1081103086=1, srv1391304479=15, srv159089925=34, srv2103717442=55, srv765023097=92, srv820456167=94, srv1616286964=35, srv367332841=66, srv825144016=95, srv155340045=27, srv1583914036=33, srv523539564=79, srv1243162198=10, srv2134096162=57, srv368952153=67, srv1125802450=2, srv1557540017=28, srv1735035005=39, srv370658453=68, srv1810352926=45, srv279633160=61, srv422647470=72, srv685559277=85, srv1196861416=6, srv1494078914=24, srv1984065800=50, srv1581055140=31, srv383029460=69, srv1497193115=25, srv1851841094=46, srv599658034=80, srv1780150796=43, srv94348159=99, srv1276893234=12, srv1394332344=16, srv737549097=89, srv1312275765=14, srv1475080100=23, srv22188=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-08 11:56:38,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-08 11:56:38,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-08 11:56:38,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-08 11:56:38,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-08 11:56:38,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-08 11:56:38,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-08 11:56:38,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-08 11:56:38,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-08 11:56:38,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-08 11:56:38,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-08 11:56:38,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-08 11:56:38,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-08 11:56:38,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-08 11:56:38,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-08 11:56:38,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-08 11:56:38,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-08 11:56:38,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-08 11:56:38,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-08 11:56:38,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-08 11:56:38,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-08 11:56:38,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-08 11:56:38,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-08 11:56:38,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-08 11:56:38,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-08 11:56:38,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-08 11:56:38,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-08 11:56:38,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-08 11:56:38,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-08 11:56:38,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-08 11:56:38,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-08 11:56:38,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-08 11:56:38,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-08 11:56:38,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-08 11:56:38,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-08 11:56:38,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-08 11:56:38,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-08 11:56:38,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-08 11:56:38,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-08 11:56:38,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-08 11:56:38,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-08 11:56:38,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-08 11:56:38,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-08 11:56:38,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-08 11:56:38,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-08 11:56:38,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-08 11:56:38,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-08 11:56:38,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-08 11:56:38,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-08 11:56:38,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-08 11:56:38,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-08 11:56:38,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-08 11:56:38,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-08 11:56:38,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-08 11:56:38,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-08 11:56:38,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-08 11:56:38,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-08 11:56:38,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-08 11:56:38,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-08 11:56:38,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-08 11:56:38,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-08 11:56:38,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-08 11:56:38,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-08 11:56:38,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-08 11:56:38,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-08 11:56:38,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-08 11:56:38,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-08 11:56:38,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-08 11:56:38,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-08 11:56:38,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-08 11:56:38,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-08 11:56:38,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-08 11:56:38,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-08 11:56:38,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-08 11:56:38,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-08 11:56:38,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-08 11:56:38,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-08 11:56:38,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-08 11:56:38,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-08 11:56:38,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-08 11:56:38,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-08 11:56:38,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-08 11:56:38,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-08 11:56:38,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-08 11:56:38,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-08 11:56:38,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-08 11:56:38,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-08 11:56:38,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-08 11:56:38,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-08 11:56:38,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-08 11:56:38,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-08 11:56:38,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-08 11:56:38,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-08 11:56:38,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-08 11:56:38,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-08 11:56:38,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-08 11:56:38,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-08 11:56:38,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-08 11:56:38,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-08 11:56:38,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-08 11:56:38,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-08 11:56:38,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-08 11:56:38,327 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:56:38,327 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-08 11:56:38,327 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-08 11:56:38,958 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 891 ms to try 5000 different iterations. Found a solution that moves 4682 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17976726931189704. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03869874954196498, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05566050692749182, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5202222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03869874954196498, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17583333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:56:38,981 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1212258769:82 , srv1450207905:82 , srv157926863:82 , srv1780150796:82 , srv666792107:82 , srv1394332344:83 , srv1616286964:83 , srv370658453:83 , srv1513178406:84 , srv1664045205:84 , srv230275016:84 , srv383029460:84 , srv757499536:84 , srv1183153101:85 , srv1391304479:85 , srv1413864525:85 , srv1475080100:85 , srv1497193115:85 , srv1685055574:85 , srv174374296:85 , srv1751846227:85 , srv1851841094:85 , srv1904948188:85 , srv1916872682:85 , srv2101810908:85 , srv22188:85 , srv279633160:85 , srv331235399:85 , srv352141811:85 , srv392485198:85 , srv765023097:85 , srv825144016:85 , srv1125802450:86 , srv1164647731:86 , srv118649404:86 , srv1243162198:86 , srv1266333902:86 , srv1312275765:86 , srv1419575123:86 , srv1473581608:86 , srv155340045:86 , srv1557540017:86 , srv1571058221:86 , srv1707073217:86 , srv1735035005:86 , srv2095545956:86 , srv358530253:86 , srv38405265:86 , srv639063953:86 , srv685559277:86 , srv69147545:86 , srv692064328:86 , srv736725489:86 , srv737549097:86 , srv770641298:86 , srv858618316:86 , srv920108541:86 , srv1494078914:87 , srv180797568:87 , srv1810352926:87 , srv2024828962:87 , srv2081605848:87 , srv2103717442:87 , srv2119807533:87 , srv2134563337:87 , srv368952153:87 , srv490868940:87 , srv61277761:87 , srv756131251:87 , srv820456167:87 , srv1081103086:88 , srv1196861416:88 , srv1276893234:88 , srv1310263082:88 , srv159089925:88 , srv1951117460:88 , srv1984065800:88 , srv367332841:88 , srv443386290:88 , srv49020945:88 , srv499487159:88 , srv516333159:88 , srv523539564:88 , srv599658034:88 , srv608406283:88 , srv94348159:88 , srv1583712728:89 , srv1741326866:89 , srv2134096162:89 , srv351682379:89 , srv1218863306:90 , srv503674964:90 , srv1219788952:91 , srv1466946618:91 , srv1581055140:92 , srv1014405857:93 , srv1583914036:93 , srv422647470:93 , srv1403179094:94 , srv889366805:434 } 2023-04-08 11:56:38,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-08 11:56:38,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1664045205=36, srv1571058221=29, srv180797568=44, srv2134563337=58, srv499487159=76, srv1219788952=9, srv2081605848=52, srv2024828962=51, srv157926863=30, srv503674964=77, srv443386290=73, srv1685055574=37, srv516333159=78, srv230275016=60, srv1310263082=13, srv2119807533=56, srv61277761=82, srv736725489=88, srv1741326866=40, srv770641298=93, srv118649404=5, srv2095545956=53, srv1466946618=21, srv756131251=90, srv1419575123=19, srv1218863306=8, srv1212258769=7, srv358530253=65, srv1904948188=47, srv666792107=84, srv174374296=41, srv1183153101=4, srv1583712728=32, srv1450207905=20, srv352141811=64, srv1473581608=22, srv1513178406=26, srv2101810908=54, srv1164647731=3, srv490868940=75, srv858618316=96, srv351682379=63, srv692064328=87, srv1916872682=48, srv920108541=98, srv757499536=91, srv639063953=83, srv1413864525=18, srv392485198=71, srv1403179094=17, srv889366805=97, srv38405265=70, srv1014405857=0, srv608406283=81, srv1707073217=38, srv1266333902=11, srv69147545=86, srv1751846227=42, srv331235399=62, srv1951117460=49, srv49020945=74, srv1081103086=1, srv1391304479=15, srv159089925=34, srv2103717442=55, srv765023097=92, srv820456167=94, srv1616286964=35, srv367332841=66, srv825144016=95, srv155340045=27, srv1583914036=33, srv523539564=79, srv1243162198=10, srv2134096162=57, srv368952153=67, srv1125802450=2, srv1557540017=28, srv1735035005=39, srv370658453=68, srv1810352926=45, srv279633160=61, srv422647470=72, srv685559277=85, srv1196861416=6, srv1494078914=24, srv1984065800=50, srv1581055140=31, srv383029460=69, srv1497193115=25, srv1851841094=46, srv599658034=80, srv1780150796=43, srv94348159=99, srv1276893234=12, srv1394332344=16, srv737549097=89, srv1312275765=14, srv1475080100=23, srv22188=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-08 11:56:39,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-08 11:56:39,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-08 11:56:39,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-08 11:56:39,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-08 11:56:39,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-08 11:56:39,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-08 11:56:39,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-08 11:56:39,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-08 11:56:39,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-08 11:56:39,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-08 11:56:39,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-08 11:56:39,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-08 11:56:39,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-08 11:56:39,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-08 11:56:39,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-08 11:56:39,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-08 11:56:39,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-08 11:56:39,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-08 11:56:39,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-08 11:56:39,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-08 11:56:39,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-08 11:56:39,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-08 11:56:39,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-08 11:56:39,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-08 11:56:39,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-08 11:56:39,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-08 11:56:39,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-08 11:56:39,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-08 11:56:39,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-08 11:56:39,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-08 11:56:39,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-08 11:56:39,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-08 11:56:39,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-08 11:56:39,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-08 11:56:39,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-08 11:56:39,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-08 11:56:39,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-08 11:56:39,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-08 11:56:39,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-08 11:56:39,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-08 11:56:39,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-08 11:56:39,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-08 11:56:39,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-08 11:56:39,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-08 11:56:39,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-08 11:56:39,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-08 11:56:39,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-08 11:56:39,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-08 11:56:39,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-08 11:56:39,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-08 11:56:39,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-08 11:56:39,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-08 11:56:39,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-08 11:56:39,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-08 11:56:39,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-08 11:56:39,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-08 11:56:39,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-08 11:56:39,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-08 11:56:39,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-08 11:56:39,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-08 11:56:39,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-08 11:56:39,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-08 11:56:39,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-08 11:56:39,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-08 11:56:39,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-08 11:56:39,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-08 11:56:39,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-08 11:56:39,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-08 11:56:39,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-08 11:56:39,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-08 11:56:39,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-08 11:56:39,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-08 11:56:39,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-08 11:56:39,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-08 11:56:39,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-08 11:56:39,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-08 11:56:39,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-08 11:56:39,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-08 11:56:39,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-08 11:56:39,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-08 11:56:39,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-08 11:56:39,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-08 11:56:39,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-08 11:56:39,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-08 11:56:39,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-08 11:56:39,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-08 11:56:39,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-08 11:56:39,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-08 11:56:39,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-08 11:56:39,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-08 11:56:39,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-08 11:56:39,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-08 11:56:39,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-08 11:56:39,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-08 11:56:39,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-08 11:56:39,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-08 11:56:39,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-08 11:56:39,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-08 11:56:39,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-08 11:56:39,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-08 11:56:39,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-08 11:56:39,183 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03869874954196498, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05566050692749182, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03869874954196498, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17583333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:56:39,183 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-08 11:56:39,183 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17973448524718919, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03869874954196498, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05566050692749182, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03869874954196498, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17583333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-08 11:56:39,472 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 417 ms to try 5000 different iterations. Found a solution that moves 2612 regions; Going from a computed imbalance of 0.17973448524718919 to a new imbalance of 0.0167688221097864. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022389382515661246); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035574489907952284, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2902222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022389382515661246); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:56:39,481 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1014405857:77 , srv1403179094:77 , srv1581055140:79 , srv1219788952:80 , srv1583914036:80 , srv422647470:80 , srv1466946618:81 , srv1218863306:82 , srv1276893234:82 , srv1310263082:82 , srv1583712728:82 , srv1741326866:82 , srv1951117460:82 , srv499487159:82 , srv503674964:82 , srv1081103086:83 , srv1196861416:83 , srv1984065800:83 , srv2134096162:83 , srv367332841:83 , srv443386290:83 , srv1164647731:84 , srv1391304479:84 , srv159089925:84 , srv180797568:84 , srv1810352926:84 , srv490868940:84 , srv516333159:84 , srv61277761:84 , srv858618316:84 , srv1125802450:85 , srv1266333902:85 , srv1413864525:85 , srv1419575123:85 , srv1707073217:85 , srv2081605848:85 , srv2119807533:85 , srv351682379:85 , srv358530253:85 , srv49020945:85 , srv599658034:85 , srv608406283:85 , srv639063953:85 , srv820456167:85 , srv118649404:86 , srv1243162198:86 , srv1473581608:86 , srv1475080100:86 , srv1494078914:86 , srv1497193115:86 , srv155340045:86 , srv1557540017:86 , srv1735035005:86 , srv1751846227:86 , srv1904948188:86 , srv2024828962:86 , srv2103717442:86 , srv22188:86 , srv279633160:86 , srv38405265:86 , srv392485198:86 , srv685559277:86 , srv737549097:86 , srv756131251:86 , srv770641298:86 , srv920108541:86 , srv1183153101:87 , srv1664045205:87 , srv1685055574:87 , srv1851841094:87 , srv1916872682:87 , srv2101810908:87 , srv2134563337:87 , srv230275016:87 , srv352141811:87 , srv523539564:87 , srv757499536:87 , srv825144016:87 , srv1312275765:88 , srv174374296:88 , srv331235399:88 , srv368952153:88 , srv370658453:88 , srv69147545:88 , srv765023097:88 , srv94348159:88 , srv1394332344:89 , srv1513178406:89 , srv1571058221:89 , srv157926863:89 , srv2095545956:89 , srv666792107:89 , srv692064328:89 , srv736725489:89 , srv1450207905:90 , srv1780150796:90 , srv1616286964:91 , srv1212258769:92 , srv383029460:92 , srv889366805:539 } 2023-04-08 11:56:39,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-08 11:56:39,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1664045205=36, srv1571058221=29, srv180797568=44, srv2134563337=58, srv499487159=76, srv1219788952=9, srv2081605848=52, srv2024828962=51, srv157926863=30, srv503674964=77, srv443386290=73, srv1685055574=37, srv516333159=78, srv230275016=60, srv1310263082=13, srv2119807533=56, srv61277761=82, srv736725489=88, srv1741326866=40, srv770641298=93, srv118649404=5, srv2095545956=53, srv1466946618=21, srv756131251=90, srv1419575123=19, srv1218863306=8, srv1212258769=7, srv358530253=65, srv1904948188=47, srv666792107=84, srv174374296=41, srv1183153101=4, srv1583712728=32, srv1450207905=20, srv352141811=64, srv1473581608=22, srv1513178406=26, srv2101810908=54, srv1164647731=3, srv490868940=75, srv858618316=96, srv351682379=63, srv692064328=87, srv1916872682=48, srv920108541=98, srv757499536=91, srv639063953=83, srv1413864525=18, srv392485198=71, srv1403179094=17, srv889366805=97, srv38405265=70, srv1014405857=0, srv608406283=81, srv1707073217=38, srv1266333902=11, srv69147545=86, srv1751846227=42, srv331235399=62, srv1951117460=49, srv49020945=74, srv1081103086=1, srv1391304479=15, srv159089925=34, srv2103717442=55, srv765023097=92, srv820456167=94, srv1616286964=35, srv367332841=66, srv825144016=95, srv155340045=27, srv1583914036=33, srv523539564=79, srv1243162198=10, srv2134096162=57, srv368952153=67, srv1125802450=2, srv1557540017=28, srv1735035005=39, srv370658453=68, srv1810352926=45, srv279633160=61, srv422647470=72, srv685559277=85, srv1196861416=6, srv1494078914=24, srv1984065800=50, srv1581055140=31, srv383029460=69, srv1497193115=25, srv1851841094=46, srv599658034=80, srv1780150796=43, srv94348159=99, srv1276893234=12, srv1394332344=16, srv737549097=89, srv1312275765=14, srv1475080100=23, srv22188=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-08 11:56:39,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-08 11:56:39,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-08 11:56:39,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-08 11:56:39,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-08 11:56:39,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-08 11:56:39,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-08 11:56:39,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-08 11:56:39,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-08 11:56:39,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-08 11:56:39,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-08 11:56:39,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-08 11:56:39,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-08 11:56:39,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-08 11:56:39,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-08 11:56:39,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-08 11:56:39,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-08 11:56:39,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-08 11:56:39,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-08 11:56:39,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-08 11:56:39,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-08 11:56:39,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-08 11:56:39,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-08 11:56:39,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-08 11:56:39,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-08 11:56:39,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-08 11:56:39,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-08 11:56:39,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-08 11:56:39,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-08 11:56:39,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-08 11:56:39,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-08 11:56:39,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-08 11:56:39,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-08 11:56:39,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-08 11:56:39,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-08 11:56:39,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-08 11:56:39,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-08 11:56:39,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-08 11:56:39,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-08 11:56:39,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-08 11:56:39,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-08 11:56:39,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-08 11:56:39,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-08 11:56:39,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-08 11:56:39,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-08 11:56:39,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-08 11:56:39,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-08 11:56:39,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-08 11:56:39,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-08 11:56:39,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-08 11:56:39,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-08 11:56:39,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-08 11:56:39,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-08 11:56:39,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-08 11:56:39,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-08 11:56:39,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-08 11:56:39,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-08 11:56:39,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-08 11:56:39,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-08 11:56:39,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-08 11:56:39,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-08 11:56:39,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-08 11:56:39,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-08 11:56:39,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-08 11:56:39,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-08 11:56:39,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-08 11:56:39,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-08 11:56:39,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-08 11:56:39,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-08 11:56:39,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-08 11:56:39,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-08 11:56:39,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-08 11:56:39,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-08 11:56:39,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-08 11:56:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-08 11:56:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-08 11:56:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-08 11:56:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-08 11:56:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-08 11:56:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-08 11:56:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-08 11:56:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-08 11:56:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-08 11:56:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-08 11:56:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-08 11:56:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-08 11:56:39,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-08 11:56:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-08 11:56:39,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-08 11:56:39,629 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022389382515661246); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035574489907952284, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022389382515661246); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:56:39,629 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-08 11:56:39,630 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016750532494874622, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022389382515661246); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035574489907952284, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022389382515661246); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-08 11:56:39,902 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 368 ms to try 5000 different iterations. Found a solution that moves 982 regions; Going from a computed imbalance of 0.016750532494874622 to a new imbalance of 3.713001222239957E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004416296951399217); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007899480725663495); MoveCostFunction : (multiplier=7.0, imbalance=0.10911111111111112, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004416296951399217); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0025833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:56:39,904 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2095545956:82 , srv368952153:82 , srv383029460:82 , srv736725489:82 , srv94348159:82 , srv1212258769:83 , srv1616286964:83 , srv2134563337:83 , srv523539564:83 , srv69147545:83 , srv1312275765:84 , srv1494078914:84 , srv1571058221:84 , srv174374296:84 , srv331235399:84 , srv351682379:84 , srv49020945:84 , srv599658034:84 , srv692064328:84 , srv1183153101:85 , srv1243162198:85 , srv1450207905:85 , srv1513178406:85 , srv155340045:85 , srv1557540017:85 , srv1583914036:85 , srv1780150796:85 , srv1851841094:85 , srv1916872682:85 , srv2024828962:85 , srv2103717442:85 , srv2119807533:85 , srv2134096162:85 , srv422647470:85 , srv503674964:85 , srv516333159:85 , srv608406283:85 , srv756131251:85 , srv765023097:85 , srv1125802450:86 , srv118649404:86 , srv1196861416:86 , srv1218863306:86 , srv1219788952:86 , srv1394332344:86 , srv1403179094:86 , srv1466946618:86 , srv1473581608:86 , srv157926863:86 , srv159089925:86 , srv1664045205:86 , srv1685055574:86 , srv1735035005:86 , srv2081605848:86 , srv2101810908:86 , srv230275016:86 , srv279633160:86 , srv352141811:86 , srv367332841:86 , srv38405265:86 , srv443386290:86 , srv490868940:86 , srv666792107:86 , srv685559277:86 , srv737549097:86 , srv770641298:86 , srv820456167:86 , srv825144016:86 , srv920108541:86 , srv1081103086:87 , srv1164647731:87 , srv1266333902:87 , srv1310263082:87 , srv1413864525:87 , srv1419575123:87 , srv1475080100:87 , srv1497193115:87 , srv1581055140:87 , srv1583712728:87 , srv1707073217:87 , srv1741326866:87 , srv1751846227:87 , srv180797568:87 , srv1810352926:87 , srv1904948188:87 , srv1951117460:87 , srv1984065800:87 , srv22188:87 , srv358530253:87 , srv370658453:87 , srv392485198:87 , srv61277761:87 , srv639063953:87 , srv757499536:87 , srv1014405857:88 , srv1276893234:88 , srv1391304479:88 , srv499487159:88 , srv858618316:88 , srv889366805:524 } 2023-04-08 11:56:39,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-08 11:56:39,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1664045205=36, srv1571058221=29, srv180797568=44, srv2134563337=58, srv499487159=76, srv1219788952=9, srv2081605848=52, srv2024828962=51, srv157926863=30, srv503674964=77, srv443386290=73, srv1685055574=37, srv516333159=78, srv230275016=60, srv1310263082=13, srv2119807533=56, srv61277761=82, srv736725489=88, srv1741326866=40, srv770641298=93, srv118649404=5, srv2095545956=53, srv1466946618=21, srv756131251=90, srv1419575123=19, srv1218863306=8, srv1212258769=7, srv358530253=65, srv1904948188=47, srv666792107=84, srv174374296=41, srv1183153101=4, srv1583712728=32, srv1450207905=20, srv352141811=64, srv1473581608=22, srv1513178406=26, srv2101810908=54, srv1164647731=3, srv490868940=75, srv858618316=96, srv351682379=63, srv692064328=87, srv1916872682=48, srv920108541=98, srv757499536=91, srv639063953=83, srv1413864525=18, srv392485198=71, srv1403179094=17, srv889366805=97, srv38405265=70, srv1014405857=0, srv608406283=81, srv1707073217=38, srv1266333902=11, srv69147545=86, srv1751846227=42, srv331235399=62, srv1951117460=49, srv49020945=74, srv1081103086=1, srv1391304479=15, srv159089925=34, srv2103717442=55, srv765023097=92, srv820456167=94, srv1616286964=35, srv367332841=66, srv825144016=95, srv155340045=27, srv1583914036=33, srv523539564=79, srv1243162198=10, srv2134096162=57, srv368952153=67, srv1125802450=2, srv1557540017=28, srv1735035005=39, srv370658453=68, srv1810352926=45, srv279633160=61, srv422647470=72, srv685559277=85, srv1196861416=6, srv1494078914=24, srv1984065800=50, srv1581055140=31, srv383029460=69, srv1497193115=25, srv1851841094=46, srv599658034=80, srv1780150796=43, srv94348159=99, srv1276893234=12, srv1394332344=16, srv737549097=89, srv1312275765=14, srv1475080100=23, srv22188=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-08 11:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-08 11:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-08 11:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-08 11:56:39,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-08 11:56:39,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-08 11:56:39,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-08 11:56:39,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-08 11:56:39,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-08 11:56:39,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-08 11:56:39,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-08 11:56:39,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-08 11:56:39,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-08 11:56:40,042 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004416296951399217); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007899480725663495); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004416296951399217); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0025833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:56:40,043 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-08 11:56:40,043 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.644240112579291E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004416296951399217); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007899480725663495); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004416296951399217); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0025833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-08 11:56:40,289 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 344 ms to try 5000 different iterations. Found a solution that moves 213 regions; Going from a computed imbalance of 3.644240112579291E-4 to a new imbalance of 2.6338957330801178E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.023666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:56:40,290 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1571058221:83 , srv1810352926:83 , srv685559277:83 , srv692064328:83 , srv1014405857:84 , srv1081103086:84 , srv1125802450:84 , srv118649404:84 , srv1218863306:84 , srv1266333902:84 , srv1276893234:84 , srv1312275765:84 , srv1394332344:84 , srv1419575123:84 , srv1466946618:84 , srv1473581608:84 , srv1475080100:84 , srv1497193115:84 , srv1513178406:84 , srv1581055140:84 , srv1583712728:84 , srv1583914036:84 , srv159089925:84 , srv1685055574:84 , srv1707073217:84 , srv1735035005:84 , srv1741326866:84 , srv174374296:84 , srv1751846227:84 , srv180797568:84 , srv1904948188:84 , srv1951117460:84 , srv1984065800:84 , srv2081605848:84 , srv2101810908:84 , srv2103717442:84 , srv2134563337:84 , srv22188:84 , srv351682379:84 , srv352141811:84 , srv358530253:84 , srv370658453:84 , srv383029460:84 , srv38405265:84 , srv392485198:84 , srv422647470:84 , srv499487159:84 , srv608406283:84 , srv61277761:84 , srv639063953:84 , srv756131251:84 , srv757499536:84 , srv765023097:84 , srv770641298:84 , srv858618316:84 , srv920108541:84 , srv94348159:84 , srv1164647731:85 , srv1183153101:85 , srv1196861416:85 , srv1212258769:85 , srv1219788952:85 , srv1243162198:85 , srv1310263082:85 , srv1391304479:85 , srv1403179094:85 , srv1413864525:85 , srv1450207905:85 , srv1494078914:85 , srv155340045:85 , srv1557540017:85 , srv157926863:85 , srv1616286964:85 , srv1664045205:85 , srv1780150796:85 , srv1851841094:85 , srv1916872682:85 , srv2024828962:85 , srv2095545956:85 , srv2119807533:85 , srv2134096162:85 , srv230275016:85 , srv279633160:85 , srv331235399:85 , srv367332841:85 , srv368952153:85 , srv443386290:85 , srv49020945:85 , srv490868940:85 , srv503674964:85 , srv516333159:85 , srv523539564:85 , srv599658034:85 , srv666792107:85 , srv736725489:85 , srv737549097:85 , srv820456167:85 , srv825144016:85 , srv69147545:86 , srv889366805:645 } 2023-04-08 11:56:40,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-08 11:56:40,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1664045205=36, srv1571058221=29, srv180797568=44, srv2134563337=58, srv499487159=76, srv1219788952=9, srv2081605848=52, srv2024828962=51, srv157926863=30, srv503674964=77, srv443386290=73, srv1685055574=37, srv516333159=78, srv230275016=60, srv1310263082=13, srv2119807533=56, srv61277761=82, srv736725489=88, srv1741326866=40, srv770641298=93, srv118649404=5, srv2095545956=53, srv1466946618=21, srv756131251=90, srv1419575123=19, srv1218863306=8, srv1212258769=7, srv358530253=65, srv1904948188=47, srv666792107=84, srv174374296=41, srv1183153101=4, srv1583712728=32, srv1450207905=20, srv352141811=64, srv1473581608=22, srv1513178406=26, srv2101810908=54, srv1164647731=3, srv490868940=75, srv858618316=96, srv351682379=63, srv692064328=87, srv1916872682=48, srv920108541=98, srv757499536=91, srv639063953=83, srv1413864525=18, srv392485198=71, srv1403179094=17, srv889366805=97, srv38405265=70, srv1014405857=0, srv608406283=81, srv1707073217=38, srv1266333902=11, srv69147545=86, srv1751846227=42, srv331235399=62, srv1951117460=49, srv49020945=74, srv1081103086=1, srv1391304479=15, srv159089925=34, srv2103717442=55, srv765023097=92, srv820456167=94, srv1616286964=35, srv367332841=66, srv825144016=95, srv155340045=27, srv1583914036=33, srv523539564=79, srv1243162198=10, srv2134096162=57, srv368952153=67, srv1125802450=2, srv1557540017=28, srv1735035005=39, srv370658453=68, srv1810352926=45, srv279633160=61, srv422647470=72, srv685559277=85, srv1196861416=6, srv1494078914=24, srv1984065800=50, srv1581055140=31, srv383029460=69, srv1497193115=25, srv1851841094=46, srv599658034=80, srv1780150796=43, srv94348159=99, srv1276893234=12, srv1394332344=16, srv737549097=89, srv1312275765=14, srv1475080100=23, srv22188=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-08 11:56:40,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-08 11:56:40,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-08 11:56:40,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-08 11:56:40,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-08 11:56:40,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-08 11:56:40,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-08 11:56:40,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-08 11:56:40,414 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:56:40,414 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-08 11:56:40,414 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.4847499453232764E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-08 11:56:40,617 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 299 ms to try 5000 different iterations. Found a solution that moves 17 regions; Going from a computed imbalance of 2.4847499453232764E-5 to a new imbalance of 7.1922758405485235E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.001888888888888889); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:56:40,618 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2024828962:83 , srv523539564:83 , srv69147545:83 , srv737549097:83 , srv820456167:83 , srv825144016:83 , srv1014405857:84 , srv1081103086:84 , srv1164647731:84 , srv1183153101:84 , srv118649404:84 , srv1196861416:84 , srv1212258769:84 , srv1218863306:84 , srv1219788952:84 , srv1243162198:84 , srv1266333902:84 , srv1276893234:84 , srv1310263082:84 , srv1312275765:84 , srv1391304479:84 , srv1394332344:84 , srv1403179094:84 , srv1413864525:84 , srv1419575123:84 , srv1450207905:84 , srv1466946618:84 , srv1473581608:84 , srv1475080100:84 , srv1494078914:84 , srv1497193115:84 , srv1513178406:84 , srv155340045:84 , srv1557540017:84 , srv1571058221:84 , srv157926863:84 , srv1581055140:84 , srv1583712728:84 , srv1583914036:84 , srv159089925:84 , srv1616286964:84 , srv1664045205:84 , srv1685055574:84 , srv1707073217:84 , srv1735035005:84 , srv1741326866:84 , srv1751846227:84 , srv1780150796:84 , srv180797568:84 , srv1851841094:84 , srv1904948188:84 , srv1916872682:84 , srv1984065800:84 , srv2081605848:84 , srv2095545956:84 , srv2101810908:84 , srv2103717442:84 , srv2119807533:84 , srv2134096162:84 , srv22188:84 , srv230275016:84 , srv279633160:84 , srv331235399:84 , srv351682379:84 , srv352141811:84 , srv358530253:84 , srv367332841:84 , srv368952153:84 , srv370658453:84 , srv383029460:84 , srv38405265:84 , srv392485198:84 , srv422647470:84 , srv443386290:84 , srv49020945:84 , srv490868940:84 , srv499487159:84 , srv503674964:84 , srv516333159:84 , srv599658034:84 , srv608406283:84 , srv61277761:84 , srv639063953:84 , srv666792107:84 , srv692064328:84 , srv736725489:84 , srv756131251:84 , srv757499536:84 , srv765023097:84 , srv770641298:84 , srv858618316:84 , srv920108541:84 , srv94348159:84 , srv1125802450:85 , srv174374296:85 , srv1810352926:85 , srv1951117460:85 , srv2134563337:85 , srv685559277:85 , srv889366805:684 } 2023-04-08 11:56:40,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-08 11:56:40,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1664045205=36, srv1571058221=29, srv180797568=44, srv2134563337=58, srv499487159=76, srv1219788952=9, srv2081605848=52, srv2024828962=51, srv157926863=30, srv503674964=77, srv443386290=73, srv1685055574=37, srv516333159=78, srv230275016=60, srv1310263082=13, srv2119807533=56, srv61277761=82, srv736725489=88, srv1741326866=40, srv770641298=93, srv118649404=5, srv2095545956=53, srv1466946618=21, srv756131251=90, srv1419575123=19, srv1218863306=8, srv1212258769=7, srv358530253=65, srv1904948188=47, srv666792107=84, srv174374296=41, srv1183153101=4, srv1583712728=32, srv1450207905=20, srv352141811=64, srv1473581608=22, srv1513178406=26, srv2101810908=54, srv1164647731=3, srv490868940=75, srv858618316=96, srv351682379=63, srv692064328=87, srv1916872682=48, srv920108541=98, srv757499536=91, srv639063953=83, srv1413864525=18, srv392485198=71, srv1403179094=17, srv889366805=97, srv38405265=70, srv1014405857=0, srv608406283=81, srv1707073217=38, srv1266333902=11, srv69147545=86, srv1751846227=42, srv331235399=62, srv1951117460=49, srv49020945=74, srv1081103086=1, srv1391304479=15, srv159089925=34, srv2103717442=55, srv765023097=92, srv820456167=94, srv1616286964=35, srv367332841=66, srv825144016=95, srv155340045=27, srv1583914036=33, srv523539564=79, srv1243162198=10, srv2134096162=57, srv368952153=67, srv1125802450=2, srv1557540017=28, srv1735035005=39, srv370658453=68, srv1810352926=45, srv279633160=61, srv422647470=72, srv685559277=85, srv1196861416=6, srv1494078914=24, srv1984065800=50, srv1581055140=31, srv383029460=69, srv1497193115=25, srv1851841094=46, srv599658034=80, srv1780150796=43, srv94348159=99, srv1276893234=12, srv1394332344=16, srv737549097=89, srv1312275765=14, srv1475080100=23, srv22188=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-08 11:56:40,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-08 11:56:40,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-08 11:56:40,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-08 11:56:40,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-08 11:56:40,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-08 11:56:40,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-08 11:56:40,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-08 11:56:40,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-08 11:56:40,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-08 11:56:40,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-08 11:56:40,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-08 11:56:40,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-08 11:56:40,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-08 11:56:40,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-08 11:56:40,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-08 11:56:40,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-08 11:56:40,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-08 11:56:40,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-08 11:56:40,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-08 11:56:40,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-08 11:56:40,741 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:56:40,741 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv2024828962:83 , srv523539564:83 , srv69147545:83 , srv737549097:83 , srv820456167:83 , srv825144016:83 , srv1014405857:84 , srv1081103086:84 , srv1164647731:84 , srv1183153101:84 , srv118649404:84 , srv1196861416:84 , srv1212258769:84 , srv1218863306:84 , srv1219788952:84 , srv1243162198:84 , srv1266333902:84 , srv1276893234:84 , srv1310263082:84 , srv1312275765:84 , srv1391304479:84 , srv1394332344:84 , srv1403179094:84 , srv1413864525:84 , srv1419575123:84 , srv1450207905:84 , srv1466946618:84 , srv1473581608:84 , srv1475080100:84 , srv1494078914:84 , srv1497193115:84 , srv1513178406:84 , srv155340045:84 , srv1557540017:84 , srv1571058221:84 , srv157926863:84 , srv1581055140:84 , srv1583712728:84 , srv1583914036:84 , srv159089925:84 , srv1616286964:84 , srv1664045205:84 , srv1685055574:84 , srv1707073217:84 , srv1735035005:84 , srv1741326866:84 , srv1751846227:84 , srv1780150796:84 , srv180797568:84 , srv1851841094:84 , srv1904948188:84 , srv1916872682:84 , srv1984065800:84 , srv2081605848:84 , srv2095545956:84 , srv2101810908:84 , srv2103717442:84 , srv2119807533:84 , srv2134096162:84 , srv22188:84 , srv230275016:84 , srv279633160:84 , srv331235399:84 , srv351682379:84 , srv352141811:84 , srv358530253:84 , srv367332841:84 , srv368952153:84 , srv370658453:84 , srv383029460:84 , srv38405265:84 , srv392485198:84 , srv422647470:84 , srv443386290:84 , srv49020945:84 , srv490868940:84 , srv499487159:84 , srv503674964:84 , srv516333159:84 , srv599658034:84 , srv608406283:84 , srv61277761:84 , srv639063953:84 , srv666792107:84 , srv692064328:84 , srv736725489:84 , srv756131251:84 , srv757499536:84 , srv765023097:84 , srv770641298:84 , srv858618316:84 , srv920108541:84 , srv94348159:84 , srv1125802450:85 , srv174374296:85 , srv1810352926:85 , srv1951117460:85 , srv2134563337:85 , srv685559277:85 , srv889366805:684 } 2023-04-08 11:56:40,765 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=421 (was 421), ProcessCount=169 (was 169), AvailableMemoryMB=2019 (was 2515) 2023-04-08 11:56:40,772 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=421, ProcessCount=169, AvailableMemoryMB=2017 2023-04-08 11:56:40,773 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-08 11:56:40,774 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-04-08 11:56:40,786 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1173568611:84 , srv123862500:84 , srv124946467:84 , srv1311499262:84 , srv1328212877:84 , srv1329550216:84 , srv1343167033:84 , srv1512084983:84 , srv153269560:84 , srv159647894:84 , srv1656528548:84 , srv1727181095:84 , srv1838533402:84 , srv191962484:84 , srv2049379797:84 , srv2065773935:84 , srv2073071491:84 , srv2114840112:84 , srv2117051935:84 , srv214507213:84 , srv247085751:84 , srv298354584:84 , srv35830378:84 , srv457752756:84 , srv607193841:84 , srv645750051:84 , srv83771987:84 , srv867876904:84 , srv957624760:84 , srv724687179:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-04-08 11:56:40,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-08 11:56:40,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1656528548=10, srv1328212877=4, srv247085751=20, srv2114840112=17, srv607193841=24, srv1512084983=7, srv2073071491=16, srv123862500=1, srv35830378=22, srv457752756=23, srv1727181095=11, srv2049379797=14, srv1311499262=3, srv1329550216=5, srv2117051935=18, srv724687179=26, srv124946467=2, srv153269560=8, srv1343167033=6, srv645750051=25, srv191962484=13, srv83771987=27, srv159647894=9, srv2065773935=15, srv957624760=29, srv867876904=28, srv1838533402=12, srv1173568611=0, srv298354584=21, srv214507213=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-08 11:56:40,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-08 11:56:40,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-08 11:56:40,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-08 11:56:40,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-08 11:56:40,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-08 11:56:40,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-08 11:56:40,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-08 11:56:40,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-08 11:56:40,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-08 11:56:40,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-08 11:56:40,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-08 11:56:40,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-08 11:56:40,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-08 11:56:40,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-08 11:56:40,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-08 11:56:40,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-08 11:56:40,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-08 11:56:40,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-08 11:56:40,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-08 11:56:40,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-08 11:56:40,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-08 11:56:40,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-08 11:56:40,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-08 11:56:40,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-08 11:56:40,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-08 11:56:40,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-08 11:56:40,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-08 11:56:40,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-08 11:56:40,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-08 11:56:40,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-08 11:56:40,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-08 11:56:40,831 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0754223139666639, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:56:40,832 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912430696512825, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0754223139666639, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-04-08 11:58:40,808 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6599856 different iterations. Found a solution that moves 1856 regions; Going from a computed imbalance of 0.9912430696512825 to a new imbalance of 4.3319965896196796E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6874074074074074, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:58:40,814 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1173568611:90 , srv123862500:90 , srv124946467:90 , srv1311499262:90 , srv1328212877:90 , srv1329550216:90 , srv1343167033:90 , srv1512084983:90 , srv153269560:90 , srv159647894:90 , srv1656528548:90 , srv1727181095:90 , srv1838533402:90 , srv191962484:90 , srv2049379797:90 , srv2065773935:90 , srv2073071491:90 , srv2114840112:90 , srv2117051935:90 , srv214507213:90 , srv247085751:90 , srv298354584:90 , srv35830378:90 , srv457752756:90 , srv607193841:90 , srv645750051:90 , srv724687179:90 , srv83771987:90 , srv867876904:90 , srv957624760:90 } 2023-04-08 11:58:40,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-08 11:58:40,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1656528548=10, srv1328212877=4, srv247085751=20, srv2114840112=17, srv607193841=24, srv1512084983=7, srv2073071491=16, srv123862500=1, srv35830378=22, srv457752756=23, srv1727181095=11, srv2049379797=14, srv1311499262=3, srv1329550216=5, srv2117051935=18, srv724687179=26, srv124946467=2, srv153269560=8, srv1343167033=6, srv645750051=25, srv191962484=13, srv83771987=27, srv159647894=9, srv2065773935=15, srv957624760=29, srv867876904=28, srv1838533402=12, srv1173568611=0, srv298354584=21, srv214507213=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-08 11:58:40,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-08 11:58:40,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-08 11:58:40,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-08 11:58:40,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-08 11:58:40,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-08 11:58:40,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-08 11:58:40,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-08 11:58:40,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-08 11:58:40,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-08 11:58:40,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-08 11:58:40,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-08 11:58:40,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-08 11:58:40,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-08 11:58:40,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-08 11:58:40,834 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-08 11:58:40,834 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1173568611:90 , srv123862500:90 , srv124946467:90 , srv1311499262:90 , srv1328212877:90 , srv1329550216:90 , srv1343167033:90 , srv1512084983:90 , srv153269560:90 , srv159647894:90 , srv1656528548:90 , srv1727181095:90 , srv1838533402:90 , srv191962484:90 , srv2049379797:90 , srv2065773935:90 , srv2073071491:90 , srv2114840112:90 , srv2117051935:90 , srv214507213:90 , srv247085751:90 , srv298354584:90 , srv35830378:90 , srv457752756:90 , srv607193841:90 , srv645750051:90 , srv724687179:90 , srv83771987:90 , srv867876904:90 , srv957624760:90 } 2023-04-08 11:58:40,848 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=270 (was 421), ProcessCount=169 (was 169), AvailableMemoryMB=2208 (was 2017) - AvailableMemoryMB LEAK? -