2023-05-10 20:53:33,011 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-10 20:53:33,340 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-10 20:53:33,461 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-10 20:53:33,523 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-10 20:53:33,532 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-10 20:53:33,544 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=292, ProcessCount=171, AvailableMemoryMB=9843 2023-05-10 20:53:33,546 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-10 20:53:33,546 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-10 20:53:33,685 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010797107:84 , srv1017570683:84 , srv1017743509:84 , srv1034736118:84 , srv1036452691:84 , srv1103904464:84 , srv1121275085:84 , srv1127984693:84 , srv1153700618:84 , srv116901489:84 , srv1172965730:84 , srv1175697426:84 , srv1185852905:84 , srv1194019595:84 , srv1241390495:84 , srv1246395021:84 , srv1252242278:84 , srv1253876279:84 , srv1256787807:84 , srv1257195917:84 , srv1272449749:84 , srv1301101033:84 , srv1358061371:84 , srv1369728755:84 , srv139648279:84 , srv1404742567:84 , srv1474039566:84 , srv1509460844:84 , srv1513154952:84 , srv1528068412:84 , srv155027060:84 , srv1570761821:84 , srv1575788913:84 , srv1596305554:84 , srv1716657996:84 , srv171674455:84 , srv1718123892:84 , srv1721480769:84 , srv1748693761:84 , srv1760096780:84 , srv1772998350:84 , srv178070484:84 , srv1793879837:84 , srv181093153:84 , srv1826022398:84 , srv1858422271:84 , srv1862829921:84 , srv1863441916:84 , srv1898993430:84 , srv1918446761:84 , srv1931557726:84 , srv1996391290:84 , srv2063557929:84 , srv209758088:84 , srv2108746958:84 , srv2118460078:84 , srv214078928:84 , srv231731022:84 , srv282949019:84 , srv290587561:84 , srv319381570:84 , srv327135737:84 , srv3378821:84 , srv338181852:84 , srv352986343:84 , srv370511549:84 , srv376723096:84 , srv387629307:84 , srv461979919:84 , srv49410684:84 , srv526510828:84 , srv545477604:84 , srv552137544:84 , srv605181101:84 , srv617399181:84 , srv62948544:84 , srv642407984:84 , srv642955976:84 , srv651375628:84 , srv660455129:84 , srv685214141:84 , srv710563956:84 , srv712347569:84 , srv713490987:84 , srv718500047:84 , srv720903511:84 , srv780085392:84 , srv807619219:84 , srv835792369:84 , srv837116009:84 , srv869938933:84 , srv880475102:84 , srv890444659:84 , srv891197784:84 , srv921023408:84 , srv939309898:84 , srv946674891:84 , srv974873559:84 , srv975878035:84 , srv1422678691:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-10 20:53:33,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-10 20:53:33,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv651375628=79, srv209758088=54, srv1772998350=41, srv1931557726=51, srv685214141=81, srv526510828=71, srv62948544=76, srv1175697426=11, srv720903511=86, srv975878035=99, srv1513154952=29, srv1252242278=16, srv1718123892=37, srv1172965730=10, srv116901489=9, srv1358061371=22, srv1862829921=47, srv807619219=88, srv552137544=73, srv1256787807=18, srv1241390495=14, srv214078928=57, srv1272449749=20, srv1509460844=28, srv1301101033=21, srv1575788913=33, srv1103904464=5, srv338181852=64, srv946674891=97, srv1034736118=3, srv181093153=44, srv642955976=78, srv1863441916=48, srv327135737=62, srv710563956=82, srv660455129=80, srv1570761821=32, srv890444659=93, srv1253876279=17, srv1760096780=40, srv1826022398=45, srv718500047=85, srv1716657996=35, srv880475102=92, srv1017570683=1, srv1246395021=15, srv376723096=67, srv319381570=61, srv1185852905=12, srv171674455=36, srv1474039566=27, srv1369728755=23, srv370511549=66, srv178070484=42, srv1858422271=46, srv461979919=69, srv1036452691=4, srv1528068412=30, srv155027060=31, srv713490987=84, srv1898993430=49, srv605181101=74, srv1996391290=52, srv2108746958=55, srv869938933=91, srv49410684=70, srv835792369=89, srv891197784=94, srv352986343=65, srv712347569=83, srv939309898=96, srv921023408=95, srv1596305554=34, srv974873559=98, srv1194019595=13, srv139648279=24, srv1257195917=19, srv1918446761=50, srv2118460078=56, srv1010797107=0, srv2063557929=53, srv837116009=90, srv1748693761=39, srv1404742567=25, srv1153700618=8, srv1017743509=2, srv1422678691=26, srv780085392=87, srv387629307=68, srv290587561=60, srv642407984=77, srv545477604=72, srv1793879837=43, srv1721480769=38, srv3378821=63, srv1127984693=7, srv1121275085=6, srv231731022=58, srv282949019=59, srv617399181=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-10 20:53:33,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-10 20:53:33,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-10 20:53:33,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-10 20:53:33,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-10 20:53:33,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-10 20:53:33,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-10 20:53:33,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-10 20:53:33,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-10 20:53:33,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-10 20:53:33,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-10 20:53:33,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-10 20:53:33,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-10 20:53:33,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-10 20:53:33,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-10 20:53:33,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-10 20:53:33,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-10 20:53:33,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-10 20:53:33,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-10 20:53:33,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-10 20:53:33,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-10 20:53:33,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-10 20:53:33,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-10 20:53:33,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-10 20:53:33,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-10 20:53:33,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-10 20:53:33,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-10 20:53:33,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-10 20:53:33,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-10 20:53:33,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-10 20:53:33,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-10 20:53:33,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-10 20:53:33,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-10 20:53:33,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-10 20:53:33,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-10 20:53:33,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-10 20:53:33,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-10 20:53:33,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-10 20:53:33,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-10 20:53:33,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-10 20:53:33,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-10 20:53:33,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-10 20:53:33,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-10 20:53:33,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-10 20:53:33,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-10 20:53:33,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-10 20:53:33,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-10 20:53:33,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-10 20:53:33,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-10 20:53:33,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-10 20:53:33,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-10 20:53:33,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-10 20:53:33,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-10 20:53:33,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-10 20:53:33,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-10 20:53:33,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-10 20:53:33,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-10 20:53:33,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-10 20:53:33,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-10 20:53:33,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-10 20:53:33,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-10 20:53:33,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-10 20:53:33,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-10 20:53:33,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-10 20:53:33,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-10 20:53:33,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-10 20:53:33,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-10 20:53:33,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-10 20:53:33,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-10 20:53:33,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-10 20:53:33,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-10 20:53:33,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-10 20:53:33,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-10 20:53:33,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-10 20:53:33,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-10 20:53:33,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-10 20:53:33,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-10 20:53:33,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-10 20:53:33,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-10 20:53:33,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-10 20:53:33,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-10 20:53:33,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-10 20:53:33,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-10 20:53:33,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-10 20:53:33,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-10 20:53:33,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-10 20:53:33,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-10 20:53:33,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-10 20:53:33,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-10 20:53:33,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-10 20:53:33,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-10 20:53:33,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-10 20:53:33,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-10 20:53:33,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-10 20:53:33,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-10 20:53:33,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-10 20:53:33,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-10 20:53:33,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-10 20:53:33,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-10 20:53:33,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-10 20:53:33,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-10 20:53:33,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-10 20:53:34,059 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:53:34,059 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-10 20:53:34,060 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-10 20:53:34,899 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1062 ms to try 5000 different iterations. Found a solution that moves 4698 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17957636311940156. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040595701421910414, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.059528638572753166, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.522, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040595701421910414, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17483333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24083333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:53:34,933 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1034736118:82 , srv1257195917:82 , srv139648279:82 , srv1748693761:82 , srv1760096780:82 , srv545477604:82 , srv1103904464:83 , srv1772998350:83 , srv1826022398:83 , srv1863441916:83 , srv461979919:83 , srv880475102:83 , srv1253876279:84 , srv1575788913:84 , srv1793879837:84 , srv370511549:84 , srv713490987:84 , srv835792369:84 , srv939309898:84 , srv1121275085:85 , srv1153700618:85 , srv1246395021:85 , srv1301101033:85 , srv1513154952:85 , srv155027060:85 , srv171674455:85 , srv181093153:85 , srv2118460078:85 , srv282949019:85 , srv387629307:85 , srv642955976:85 , srv718500047:85 , srv921023408:85 , srv946674891:85 , srv1017570683:86 , srv116901489:86 , srv1194019595:86 , srv1252242278:86 , srv1272449749:86 , srv1358061371:86 , srv1721480769:86 , srv1898993430:86 , srv1931557726:86 , srv209758088:86 , srv290587561:86 , srv319381570:86 , srv49410684:86 , srv526510828:86 , srv552137544:86 , srv62948544:86 , srv685214141:86 , srv780085392:86 , srv807619219:86 , srv837116009:86 , srv890444659:86 , srv891197784:86 , srv1474039566:87 , srv1528068412:87 , srv1596305554:87 , srv1718123892:87 , srv178070484:87 , srv338181852:87 , srv720903511:87 , srv869938933:87 , srv975878035:87 , srv1127984693:88 , srv1172965730:88 , srv1185852905:88 , srv1241390495:88 , srv1404742567:88 , srv1509460844:88 , srv1570761821:88 , srv1862829921:88 , srv1918446761:88 , srv1996391290:88 , srv327135737:88 , srv3378821:88 , srv352986343:88 , srv376723096:88 , srv617399181:88 , srv660455129:88 , srv1010797107:89 , srv1036452691:89 , srv1175697426:89 , srv1256787807:89 , srv1716657996:89 , srv1858422271:89 , srv2063557929:89 , srv2108746958:89 , srv214078928:89 , srv642407984:89 , srv651375628:89 , srv710563956:89 , srv974873559:89 , srv1017743509:90 , srv231731022:90 , srv1369728755:91 , srv712347569:91 , srv605181101:94 , srv1422678691:451 } 2023-05-10 20:53:34,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-10 20:53:34,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv651375628=79, srv209758088=54, srv1772998350=41, srv1931557726=51, srv685214141=81, srv526510828=71, srv62948544=76, srv1175697426=11, srv720903511=86, srv975878035=99, srv1513154952=29, srv1252242278=16, srv1718123892=37, srv1172965730=10, srv116901489=9, srv1358061371=22, srv1862829921=47, srv807619219=88, srv552137544=73, srv1256787807=18, srv1241390495=14, srv214078928=57, srv1272449749=20, srv1509460844=28, srv1301101033=21, srv1575788913=33, srv1103904464=5, srv338181852=64, srv946674891=97, srv1034736118=3, srv181093153=44, srv642955976=78, srv1863441916=48, srv327135737=62, srv710563956=82, srv660455129=80, srv1570761821=32, srv890444659=93, srv1253876279=17, srv1760096780=40, srv1826022398=45, srv718500047=85, srv1716657996=35, srv880475102=92, srv1017570683=1, srv1246395021=15, srv376723096=67, srv319381570=61, srv1185852905=12, srv171674455=36, srv1474039566=27, srv1369728755=23, srv370511549=66, srv178070484=42, srv1858422271=46, srv461979919=69, srv1036452691=4, srv1528068412=30, srv155027060=31, srv713490987=84, srv1898993430=49, srv605181101=74, srv1996391290=52, srv2108746958=55, srv869938933=91, srv49410684=70, srv835792369=89, srv891197784=94, srv352986343=65, srv712347569=83, srv939309898=96, srv921023408=95, srv1596305554=34, srv974873559=98, srv1194019595=13, srv139648279=24, srv1257195917=19, srv1918446761=50, srv2118460078=56, srv1010797107=0, srv2063557929=53, srv837116009=90, srv1748693761=39, srv1404742567=25, srv1153700618=8, srv1017743509=2, srv1422678691=26, srv780085392=87, srv387629307=68, srv290587561=60, srv642407984=77, srv545477604=72, srv1793879837=43, srv1721480769=38, srv3378821=63, srv1127984693=7, srv1121275085=6, srv231731022=58, srv282949019=59, srv617399181=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-10 20:53:34,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-10 20:53:34,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-10 20:53:34,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-10 20:53:34,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-10 20:53:34,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-10 20:53:34,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-10 20:53:34,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-10 20:53:34,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-10 20:53:34,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-10 20:53:34,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-10 20:53:34,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-10 20:53:34,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-10 20:53:34,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-10 20:53:34,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-10 20:53:34,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-10 20:53:34,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-10 20:53:34,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-10 20:53:34,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-10 20:53:34,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-10 20:53:34,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-10 20:53:34,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-10 20:53:34,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-10 20:53:34,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-10 20:53:34,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-10 20:53:34,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-10 20:53:34,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-10 20:53:34,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-10 20:53:34,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-10 20:53:34,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-10 20:53:34,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-10 20:53:34,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-10 20:53:34,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-10 20:53:34,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-10 20:53:34,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-10 20:53:34,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-10 20:53:34,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-10 20:53:34,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-10 20:53:34,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-10 20:53:34,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-10 20:53:34,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-10 20:53:34,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-10 20:53:34,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-10 20:53:34,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-10 20:53:34,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-10 20:53:34,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-10 20:53:34,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-10 20:53:34,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-10 20:53:34,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-10 20:53:34,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-10 20:53:34,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-10 20:53:34,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-10 20:53:34,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-10 20:53:34,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-10 20:53:34,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-10 20:53:34,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-10 20:53:34,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-10 20:53:34,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-10 20:53:34,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-10 20:53:34,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-10 20:53:34,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-10 20:53:34,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-10 20:53:34,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-10 20:53:34,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-10 20:53:34,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-10 20:53:34,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-10 20:53:34,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-10 20:53:34,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-10 20:53:34,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-10 20:53:34,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-10 20:53:34,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-10 20:53:34,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-10 20:53:34,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-10 20:53:34,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-10 20:53:34,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-10 20:53:34,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-10 20:53:34,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-10 20:53:34,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-10 20:53:34,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-10 20:53:34,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-10 20:53:34,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-10 20:53:34,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-10 20:53:34,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-10 20:53:34,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-10 20:53:34,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-10 20:53:34,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-10 20:53:34,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-10 20:53:34,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-10 20:53:34,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-10 20:53:34,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-10 20:53:34,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-10 20:53:34,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-10 20:53:34,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-10 20:53:34,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-10 20:53:34,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-10 20:53:34,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-10 20:53:34,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-10 20:53:34,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-10 20:53:34,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-10 20:53:34,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-10 20:53:34,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-10 20:53:34,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-10 20:53:35,170 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040595701421910414, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.059528638572753166, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040595701421910414, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17483333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24083333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:53:35,171 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-10 20:53:35,171 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17954346702029914, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040595701421910414, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.059528638572753166, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040595701421910414, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17483333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24083333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-10 20:53:35,509 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 507 ms to try 5000 different iterations. Found a solution that moves 2581 regions; Going from a computed imbalance of 0.17954346702029914 to a new imbalance of 0.014623753190162542. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023397112341983238); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03800880044589119, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2867777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023397112341983238); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04741666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:53:35,523 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv605181101:78 , srv1369728755:80 , srv1010797107:81 , srv1017743509:81 , srv1858422271:81 , srv231731022:81 , srv712347569:81 , srv1716657996:82 , srv2108746958:82 , srv352986343:82 , srv376723096:82 , srv975878035:82 , srv1172965730:83 , srv1175697426:83 , srv1256787807:83 , srv1404742567:83 , srv2063557929:83 , srv3378821:83 , srv617399181:83 , srv660455129:83 , srv974873559:83 , srv1017570683:84 , srv1127984693:84 , srv1241390495:84 , srv1509460844:84 , srv1918446761:84 , srv685214141:84 , srv780085392:84 , srv807619219:84 , srv1474039566:85 , srv1570761821:85 , srv1596305554:85 , srv1718123892:85 , srv1862829921:85 , srv1996391290:85 , srv2118460078:85 , srv214078928:85 , srv338181852:85 , srv552137544:85 , srv651375628:85 , srv720903511:85 , srv869938933:85 , srv891197784:85 , srv1036452691:86 , srv1121275085:86 , srv116901489:86 , srv1185852905:86 , srv1194019595:86 , srv1246395021:86 , srv1252242278:86 , srv1272449749:86 , srv1301101033:86 , srv1358061371:86 , srv155027060:86 , srv1575788913:86 , srv171674455:86 , srv209758088:86 , srv282949019:86 , srv327135737:86 , srv370511549:86 , srv49410684:86 , srv62948544:86 , srv710563956:86 , srv718500047:86 , srv890444659:86 , srv921023408:86 , srv1748693761:87 , srv1898993430:87 , srv1931557726:87 , srv387629307:87 , srv526510828:87 , srv642407984:87 , srv642955976:87 , srv837116009:87 , srv880475102:87 , srv939309898:87 , srv1153700618:88 , srv1513154952:88 , srv1528068412:88 , srv178070484:88 , srv1793879837:88 , srv181093153:88 , srv1863441916:88 , srv290587561:88 , srv461979919:88 , srv835792369:88 , srv946674891:88 , srv1034736118:89 , srv1253876279:89 , srv1257195917:89 , srv1721480769:89 , srv1760096780:89 , srv1772998350:89 , srv1826022398:89 , srv319381570:89 , srv545477604:89 , srv713490987:89 , srv1103904464:90 , srv139648279:91 , srv1422678691:531 } 2023-05-10 20:53:35,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-10 20:53:35,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv651375628=79, srv209758088=54, srv1772998350=41, srv1931557726=51, srv685214141=81, srv526510828=71, srv62948544=76, srv1175697426=11, srv720903511=86, srv975878035=99, srv1513154952=29, srv1252242278=16, srv1718123892=37, srv1172965730=10, srv116901489=9, srv1358061371=22, srv1862829921=47, srv807619219=88, srv552137544=73, srv1256787807=18, srv1241390495=14, srv214078928=57, srv1272449749=20, srv1509460844=28, srv1301101033=21, srv1575788913=33, srv1103904464=5, srv338181852=64, srv946674891=97, srv1034736118=3, srv181093153=44, srv642955976=78, srv1863441916=48, srv327135737=62, srv710563956=82, srv660455129=80, srv1570761821=32, srv890444659=93, srv1253876279=17, srv1760096780=40, srv1826022398=45, srv718500047=85, srv1716657996=35, srv880475102=92, srv1017570683=1, srv1246395021=15, srv376723096=67, srv319381570=61, srv1185852905=12, srv171674455=36, srv1474039566=27, srv1369728755=23, srv370511549=66, srv178070484=42, srv1858422271=46, srv461979919=69, srv1036452691=4, srv1528068412=30, srv155027060=31, srv713490987=84, srv1898993430=49, srv605181101=74, srv1996391290=52, srv2108746958=55, srv869938933=91, srv49410684=70, srv835792369=89, srv891197784=94, srv352986343=65, srv712347569=83, srv939309898=96, srv921023408=95, srv1596305554=34, srv974873559=98, srv1194019595=13, srv139648279=24, srv1257195917=19, srv1918446761=50, srv2118460078=56, srv1010797107=0, srv2063557929=53, srv837116009=90, srv1748693761=39, srv1404742567=25, srv1153700618=8, srv1017743509=2, srv1422678691=26, srv780085392=87, srv387629307=68, srv290587561=60, srv642407984=77, srv545477604=72, srv1793879837=43, srv1721480769=38, srv3378821=63, srv1127984693=7, srv1121275085=6, srv231731022=58, srv282949019=59, srv617399181=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-10 20:53:35,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-10 20:53:35,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-10 20:53:35,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-10 20:53:35,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-10 20:53:35,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-10 20:53:35,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-10 20:53:35,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-10 20:53:35,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-10 20:53:35,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-10 20:53:35,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-10 20:53:35,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-10 20:53:35,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-10 20:53:35,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-10 20:53:35,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-10 20:53:35,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-10 20:53:35,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-10 20:53:35,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-10 20:53:35,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-10 20:53:35,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-10 20:53:35,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-10 20:53:35,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-10 20:53:35,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-10 20:53:35,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-10 20:53:35,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-10 20:53:35,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-10 20:53:35,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-10 20:53:35,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-10 20:53:35,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-10 20:53:35,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-10 20:53:35,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-10 20:53:35,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-10 20:53:35,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-10 20:53:35,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-10 20:53:35,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-10 20:53:35,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-10 20:53:35,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-10 20:53:35,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-10 20:53:35,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-10 20:53:35,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-10 20:53:35,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-10 20:53:35,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-10 20:53:35,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-10 20:53:35,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-10 20:53:35,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-10 20:53:35,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-10 20:53:35,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-10 20:53:35,695 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023397112341983238); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03800880044589119, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023397112341983238); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04741666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:53:35,695 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-10 20:53:35,695 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014605680641890223, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023397112341983238); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03800880044589119, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023397112341983238); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04741666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-10 20:53:35,951 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 371 ms to try 5000 different iterations. Found a solution that moves 968 regions; Going from a computed imbalance of 0.014605680641890223 to a new imbalance of 7.88215868283949E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0072163923634310095); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009211323729436767); MoveCostFunction : (multiplier=7.0, imbalance=0.10755555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0072163923634310095); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:53:35,953 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1036452691:82 , srv1528068412:82 , srv1721480769:82 , srv178070484:82 , srv642407984:82 , srv319381570:83 , srv327135737:83 , srv710563956:83 , srv1103904464:84 , srv1153700618:84 , srv116901489:84 , srv1185852905:84 , srv1253876279:84 , srv1358061371:84 , srv1570761821:84 , srv181093153:84 , srv1862829921:84 , srv1898993430:84 , srv1931557726:84 , srv1996391290:84 , srv214078928:84 , srv290587561:84 , srv526510828:84 , srv651375628:84 , srv946674891:84 , srv1127984693:85 , srv1241390495:85 , srv1252242278:85 , srv1256787807:85 , srv1272449749:85 , srv139648279:85 , srv1509460844:85 , srv1513154952:85 , srv1596305554:85 , srv1772998350:85 , srv1826022398:85 , srv2063557929:85 , srv209758088:85 , srv282949019:85 , srv338181852:85 , srv49410684:85 , srv552137544:85 , srv62948544:85 , srv642955976:85 , srv712347569:85 , srv713490987:85 , srv720903511:85 , srv837116009:85 , srv890444659:85 , srv974873559:85 , srv1017743509:86 , srv1034736118:86 , srv1121275085:86 , srv1172965730:86 , srv1175697426:86 , srv1194019595:86 , srv1301101033:86 , srv1369728755:86 , srv1404742567:86 , srv1474039566:86 , srv155027060:86 , srv171674455:86 , srv1718123892:86 , srv1760096780:86 , srv1863441916:86 , srv1918446761:86 , srv2108746958:86 , srv231731022:86 , srv3378821:86 , srv387629307:86 , srv461979919:86 , srv545477604:86 , srv605181101:86 , srv660455129:86 , srv718500047:86 , srv835792369:86 , srv869938933:86 , srv1246395021:87 , srv1716657996:87 , srv1748693761:87 , srv1793879837:87 , srv370511549:87 , srv685214141:87 , srv780085392:87 , srv880475102:87 , srv891197784:87 , srv921023408:87 , srv939309898:87 , srv1010797107:88 , srv1017570683:88 , srv1257195917:88 , srv1575788913:88 , srv1858422271:88 , srv2118460078:88 , srv352986343:88 , srv376723096:88 , srv617399181:88 , srv807619219:88 , srv975878035:89 , srv1422678691:540 } 2023-05-10 20:53:35,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-10 20:53:35,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv651375628=79, srv209758088=54, srv1772998350=41, srv1931557726=51, srv685214141=81, srv526510828=71, srv62948544=76, srv1175697426=11, srv720903511=86, srv975878035=99, srv1513154952=29, srv1252242278=16, srv1718123892=37, srv1172965730=10, srv116901489=9, srv1358061371=22, srv1862829921=47, srv807619219=88, srv552137544=73, srv1256787807=18, srv1241390495=14, srv214078928=57, srv1272449749=20, srv1509460844=28, srv1301101033=21, srv1575788913=33, srv1103904464=5, srv338181852=64, srv946674891=97, srv1034736118=3, srv181093153=44, srv642955976=78, srv1863441916=48, srv327135737=62, srv710563956=82, srv660455129=80, srv1570761821=32, srv890444659=93, srv1253876279=17, srv1760096780=40, srv1826022398=45, srv718500047=85, srv1716657996=35, srv880475102=92, srv1017570683=1, srv1246395021=15, srv376723096=67, srv319381570=61, srv1185852905=12, srv171674455=36, srv1474039566=27, srv1369728755=23, srv370511549=66, srv178070484=42, srv1858422271=46, srv461979919=69, srv1036452691=4, srv1528068412=30, srv155027060=31, srv713490987=84, srv1898993430=49, srv605181101=74, srv1996391290=52, srv2108746958=55, srv869938933=91, srv49410684=70, srv835792369=89, srv891197784=94, srv352986343=65, srv712347569=83, srv939309898=96, srv921023408=95, srv1596305554=34, srv974873559=98, srv1194019595=13, srv139648279=24, srv1257195917=19, srv1918446761=50, srv2118460078=56, srv1010797107=0, srv2063557929=53, srv837116009=90, srv1748693761=39, srv1404742567=25, srv1153700618=8, srv1017743509=2, srv1422678691=26, srv780085392=87, srv387629307=68, srv290587561=60, srv642407984=77, srv545477604=72, srv1793879837=43, srv1721480769=38, srv3378821=63, srv1127984693=7, srv1121275085=6, srv231731022=58, srv282949019=59, srv617399181=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-10 20:53:35,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-10 20:53:35,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-10 20:53:35,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-10 20:53:35,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-10 20:53:35,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-10 20:53:35,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-10 20:53:35,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-10 20:53:35,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-10 20:53:35,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-10 20:53:35,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-10 20:53:35,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-10 20:53:35,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-10 20:53:35,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-10 20:53:35,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-10 20:53:35,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-10 20:53:35,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-10 20:53:35,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-10 20:53:35,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-10 20:53:35,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-10 20:53:35,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-10 20:53:35,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-10 20:53:35,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-10 20:53:35,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-10 20:53:35,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-10 20:53:35,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-10 20:53:35,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-10 20:53:35,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-10 20:53:35,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-10 20:53:35,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-10 20:53:35,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-10 20:53:35,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-10 20:53:35,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-10 20:53:35,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-10 20:53:35,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-10 20:53:35,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-10 20:53:35,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-10 20:53:35,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-10 20:53:35,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-10 20:53:35,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-10 20:53:35,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-10 20:53:35,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-10 20:53:35,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-10 20:53:35,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-10 20:53:35,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-10 20:53:35,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-10 20:53:35,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-10 20:53:35,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-10 20:53:35,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-10 20:53:35,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-10 20:53:35,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-10 20:53:35,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-10 20:53:35,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-10 20:53:35,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-10 20:53:35,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-10 20:53:35,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-10 20:53:35,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-10 20:53:35,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-10 20:53:35,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-10 20:53:35,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-10 20:53:35,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-10 20:53:35,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-10 20:53:35,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-10 20:53:35,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-10 20:53:35,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-10 20:53:35,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-10 20:53:35,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-10 20:53:35,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-10 20:53:35,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-10 20:53:35,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-10 20:53:35,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-10 20:53:35,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-10 20:53:35,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-10 20:53:35,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-10 20:53:35,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-10 20:53:35,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-10 20:53:35,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-10 20:53:35,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-10 20:53:35,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-10 20:53:35,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-10 20:53:35,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-10 20:53:35,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-10 20:53:35,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-10 20:53:35,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-10 20:53:35,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-10 20:53:35,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-10 20:53:35,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-10 20:53:35,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-10 20:53:35,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-10 20:53:36,110 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0072163923634310095); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009211323729436767); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0072163923634310095); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:53:36,110 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-10 20:53:36,110 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.814377874131216E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0072163923634310095); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009211323729436767); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0072163923634310095); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-10 20:53:36,451 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 455 ms to try 5000 different iterations. Found a solution that moves 256 regions; Going from a computed imbalance of 7.814377874131216E-4 to a new imbalance of 4.2713084172899956E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.028444444444444446, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:53:36,452 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1257195917:83 , srv387629307:83 , srv605181101:83 , srv1010797107:84 , srv1017570683:84 , srv1036452691:84 , srv1175697426:84 , srv1185852905:84 , srv1194019595:84 , srv1241390495:84 , srv1246395021:84 , srv139648279:84 , srv1474039566:84 , srv1513154952:84 , srv1575788913:84 , srv1596305554:84 , srv171674455:84 , srv1718123892:84 , srv1793879837:84 , srv1858422271:84 , srv1918446761:84 , srv2118460078:84 , srv214078928:84 , srv290587561:84 , srv319381570:84 , srv352986343:84 , srv376723096:84 , srv617399181:84 , srv642407984:84 , srv651375628:84 , srv710563956:84 , srv713490987:84 , srv807619219:84 , srv835792369:84 , srv837116009:84 , srv869938933:84 , srv891197784:84 , srv921023408:84 , srv939309898:84 , srv975878035:84 , srv1017743509:85 , srv1034736118:85 , srv1103904464:85 , srv1121275085:85 , srv1127984693:85 , srv1153700618:85 , srv1172965730:85 , srv1252242278:85 , srv1253876279:85 , srv1256787807:85 , srv1272449749:85 , srv1301101033:85 , srv1369728755:85 , srv1404742567:85 , srv1509460844:85 , srv1528068412:85 , srv155027060:85 , srv1570761821:85 , srv1716657996:85 , srv1721480769:85 , srv1760096780:85 , srv1772998350:85 , srv178070484:85 , srv1826022398:85 , srv1862829921:85 , srv1863441916:85 , srv1898993430:85 , srv1931557726:85 , srv1996391290:85 , srv2063557929:85 , srv209758088:85 , srv2108746958:85 , srv231731022:85 , srv327135737:85 , srv3378821:85 , srv338181852:85 , srv370511549:85 , srv461979919:85 , srv49410684:85 , srv526510828:85 , srv545477604:85 , srv62948544:85 , srv642955976:85 , srv660455129:85 , srv685214141:85 , srv718500047:85 , srv720903511:85 , srv780085392:85 , srv880475102:85 , srv890444659:85 , srv946674891:85 , srv974873559:85 , srv116901489:86 , srv1358061371:86 , srv1748693761:86 , srv181093153:86 , srv282949019:86 , srv712347569:86 , srv552137544:87 , srv1422678691:620 } 2023-05-10 20:53:36,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-10 20:53:36,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv651375628=79, srv209758088=54, srv1772998350=41, srv1931557726=51, srv685214141=81, srv526510828=71, srv62948544=76, srv1175697426=11, srv720903511=86, srv975878035=99, srv1513154952=29, srv1252242278=16, srv1718123892=37, srv1172965730=10, srv116901489=9, srv1358061371=22, srv1862829921=47, srv807619219=88, srv552137544=73, srv1256787807=18, srv1241390495=14, srv214078928=57, srv1272449749=20, srv1509460844=28, srv1301101033=21, srv1575788913=33, srv1103904464=5, srv338181852=64, srv946674891=97, srv1034736118=3, srv181093153=44, srv642955976=78, srv1863441916=48, srv327135737=62, srv710563956=82, srv660455129=80, srv1570761821=32, srv890444659=93, srv1253876279=17, srv1760096780=40, srv1826022398=45, srv718500047=85, srv1716657996=35, srv880475102=92, srv1017570683=1, srv1246395021=15, srv376723096=67, srv319381570=61, srv1185852905=12, srv171674455=36, srv1474039566=27, srv1369728755=23, srv370511549=66, srv178070484=42, srv1858422271=46, srv461979919=69, srv1036452691=4, srv1528068412=30, srv155027060=31, srv713490987=84, srv1898993430=49, srv605181101=74, srv1996391290=52, srv2108746958=55, srv869938933=91, srv49410684=70, srv835792369=89, srv891197784=94, srv352986343=65, srv712347569=83, srv939309898=96, srv921023408=95, srv1596305554=34, srv974873559=98, srv1194019595=13, srv139648279=24, srv1257195917=19, srv1918446761=50, srv2118460078=56, srv1010797107=0, srv2063557929=53, srv837116009=90, srv1748693761=39, srv1404742567=25, srv1153700618=8, srv1017743509=2, srv1422678691=26, srv780085392=87, srv387629307=68, srv290587561=60, srv642407984=77, srv545477604=72, srv1793879837=43, srv1721480769=38, srv3378821=63, srv1127984693=7, srv1121275085=6, srv231731022=58, srv282949019=59, srv617399181=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-10 20:53:36,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-10 20:53:36,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-10 20:53:36,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-10 20:53:36,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-10 20:53:36,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-10 20:53:36,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-10 20:53:36,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-10 20:53:36,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-10 20:53:36,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-10 20:53:36,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-10 20:53:36,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-10 20:53:36,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-10 20:53:36,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-10 20:53:36,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-10 20:53:36,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-10 20:53:36,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-10 20:53:36,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-10 20:53:36,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-10 20:53:36,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-10 20:53:36,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-10 20:53:36,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-10 20:53:36,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-10 20:53:36,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-10 20:53:36,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-10 20:53:36,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-10 20:53:36,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-10 20:53:36,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-10 20:53:36,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-10 20:53:36,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-10 20:53:36,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-10 20:53:36,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-10 20:53:36,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-10 20:53:36,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-10 20:53:36,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-10 20:53:36,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-10 20:53:36,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-10 20:53:36,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-10 20:53:36,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-10 20:53:36,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-10 20:53:36,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-10 20:53:36,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-10 20:53:36,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-10 20:53:36,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-10 20:53:36,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-10 20:53:36,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-10 20:53:36,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-10 20:53:36,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-10 20:53:36,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-10 20:53:36,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-10 20:53:36,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-10 20:53:36,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-10 20:53:36,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-10 20:53:36,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-10 20:53:36,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-10 20:53:36,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-10 20:53:36,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-10 20:53:36,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-10 20:53:36,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-10 20:53:36,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-10 20:53:36,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-10 20:53:36,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-10 20:53:36,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-10 20:53:36,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-10 20:53:36,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-10 20:53:36,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-10 20:53:36,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-10 20:53:36,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-10 20:53:36,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-10 20:53:36,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-10 20:53:36,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-10 20:53:36,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-10 20:53:36,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-10 20:53:36,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-10 20:53:36,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-10 20:53:36,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-10 20:53:36,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-10 20:53:36,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-10 20:53:36,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-10 20:53:36,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-10 20:53:36,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-10 20:53:36,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-10 20:53:36,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-10 20:53:36,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-10 20:53:36,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-10 20:53:36,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-10 20:53:36,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-10 20:53:36,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-10 20:53:36,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-10 20:53:36,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-10 20:53:36,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-10 20:53:36,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-10 20:53:36,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-10 20:53:36,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-10 20:53:36,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-10 20:53:36,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-10 20:53:36,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-10 20:53:36,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-10 20:53:36,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-10 20:53:36,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-10 20:53:36,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-10 20:53:36,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-10 20:53:36,596 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:53:36,597 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-10 20:53:36,597 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.092053385995388E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-10 20:53:36,846 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 358 ms to try 5000 different iterations. Found a solution that moves 24 regions; Going from a computed imbalance of 4.092053385995388E-5 to a new imbalance of 7.555809092151254E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0026666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:53:36,847 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1716657996:83 , srv1793879837:83 , srv181093153:83 , srv552137544:83 , srv617399181:83 , srv712347569:83 , srv1010797107:84 , srv1017570683:84 , srv1017743509:84 , srv1034736118:84 , srv1103904464:84 , srv1121275085:84 , srv1127984693:84 , srv1153700618:84 , srv116901489:84 , srv1172965730:84 , srv1175697426:84 , srv1185852905:84 , srv1194019595:84 , srv1246395021:84 , srv1252242278:84 , srv1253876279:84 , srv1256787807:84 , srv1257195917:84 , srv1272449749:84 , srv1301101033:84 , srv1358061371:84 , srv1369728755:84 , srv139648279:84 , srv1404742567:84 , srv1474039566:84 , srv1509460844:84 , srv1513154952:84 , srv1528068412:84 , srv155027060:84 , srv1570761821:84 , srv1575788913:84 , srv1718123892:84 , srv1721480769:84 , srv1748693761:84 , srv1760096780:84 , srv1772998350:84 , srv178070484:84 , srv1826022398:84 , srv1858422271:84 , srv1862829921:84 , srv1863441916:84 , srv1898993430:84 , srv1918446761:84 , srv1931557726:84 , srv1996391290:84 , srv2063557929:84 , srv209758088:84 , srv2108746958:84 , srv2118460078:84 , srv214078928:84 , srv231731022:84 , srv282949019:84 , srv290587561:84 , srv319381570:84 , srv327135737:84 , srv3378821:84 , srv338181852:84 , srv352986343:84 , srv370511549:84 , srv376723096:84 , srv461979919:84 , srv49410684:84 , srv526510828:84 , srv545477604:84 , srv62948544:84 , srv642407984:84 , srv642955976:84 , srv651375628:84 , srv660455129:84 , srv685214141:84 , srv710563956:84 , srv713490987:84 , srv718500047:84 , srv720903511:84 , srv780085392:84 , srv807619219:84 , srv835792369:84 , srv837116009:84 , srv869938933:84 , srv880475102:84 , srv890444659:84 , srv891197784:84 , srv921023408:84 , srv939309898:84 , srv946674891:84 , srv974873559:84 , srv975878035:84 , srv1036452691:85 , srv1241390495:85 , srv1596305554:85 , srv171674455:85 , srv387629307:85 , srv605181101:85 , srv1422678691:684 } 2023-05-10 20:53:36,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-10 20:53:36,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv651375628=79, srv209758088=54, srv1772998350=41, srv1931557726=51, srv685214141=81, srv526510828=71, srv62948544=76, srv1175697426=11, srv720903511=86, srv975878035=99, srv1513154952=29, srv1252242278=16, srv1718123892=37, srv1172965730=10, srv116901489=9, srv1358061371=22, srv1862829921=47, srv807619219=88, srv552137544=73, srv1256787807=18, srv1241390495=14, srv214078928=57, srv1272449749=20, srv1509460844=28, srv1301101033=21, srv1575788913=33, srv1103904464=5, srv338181852=64, srv946674891=97, srv1034736118=3, srv181093153=44, srv642955976=78, srv1863441916=48, srv327135737=62, srv710563956=82, srv660455129=80, srv1570761821=32, srv890444659=93, srv1253876279=17, srv1760096780=40, srv1826022398=45, srv718500047=85, srv1716657996=35, srv880475102=92, srv1017570683=1, srv1246395021=15, srv376723096=67, srv319381570=61, srv1185852905=12, srv171674455=36, srv1474039566=27, srv1369728755=23, srv370511549=66, srv178070484=42, srv1858422271=46, srv461979919=69, srv1036452691=4, srv1528068412=30, srv155027060=31, srv713490987=84, srv1898993430=49, srv605181101=74, srv1996391290=52, srv2108746958=55, srv869938933=91, srv49410684=70, srv835792369=89, srv891197784=94, srv352986343=65, srv712347569=83, srv939309898=96, srv921023408=95, srv1596305554=34, srv974873559=98, srv1194019595=13, srv139648279=24, srv1257195917=19, srv1918446761=50, srv2118460078=56, srv1010797107=0, srv2063557929=53, srv837116009=90, srv1748693761=39, srv1404742567=25, srv1153700618=8, srv1017743509=2, srv1422678691=26, srv780085392=87, srv387629307=68, srv290587561=60, srv642407984=77, srv545477604=72, srv1793879837=43, srv1721480769=38, srv3378821=63, srv1127984693=7, srv1121275085=6, srv231731022=58, srv282949019=59, srv617399181=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-10 20:53:36,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-10 20:53:36,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-10 20:53:36,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-10 20:53:36,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-10 20:53:36,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-10 20:53:36,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-10 20:53:36,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-10 20:53:36,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-10 20:53:36,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-10 20:53:36,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-10 20:53:36,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-10 20:53:36,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-10 20:53:36,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-10 20:53:36,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-10 20:53:36,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-10 20:53:36,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-10 20:53:36,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-10 20:53:36,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-10 20:53:36,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-10 20:53:36,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-10 20:53:36,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-10 20:53:36,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-10 20:53:36,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-10 20:53:36,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-10 20:53:36,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-10 20:53:36,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-10 20:53:36,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-10 20:53:36,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-10 20:53:36,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-10 20:53:36,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-10 20:53:36,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-10 20:53:36,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-10 20:53:36,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-10 20:53:37,017 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:53:37,018 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1716657996:83 , srv1793879837:83 , srv181093153:83 , srv552137544:83 , srv617399181:83 , srv712347569:83 , srv1010797107:84 , srv1017570683:84 , srv1017743509:84 , srv1034736118:84 , srv1103904464:84 , srv1121275085:84 , srv1127984693:84 , srv1153700618:84 , srv116901489:84 , srv1172965730:84 , srv1175697426:84 , srv1185852905:84 , srv1194019595:84 , srv1246395021:84 , srv1252242278:84 , srv1253876279:84 , srv1256787807:84 , srv1257195917:84 , srv1272449749:84 , srv1301101033:84 , srv1358061371:84 , srv1369728755:84 , srv139648279:84 , srv1404742567:84 , srv1474039566:84 , srv1509460844:84 , srv1513154952:84 , srv1528068412:84 , srv155027060:84 , srv1570761821:84 , srv1575788913:84 , srv1718123892:84 , srv1721480769:84 , srv1748693761:84 , srv1760096780:84 , srv1772998350:84 , srv178070484:84 , srv1826022398:84 , srv1858422271:84 , srv1862829921:84 , srv1863441916:84 , srv1898993430:84 , srv1918446761:84 , srv1931557726:84 , srv1996391290:84 , srv2063557929:84 , srv209758088:84 , srv2108746958:84 , srv2118460078:84 , srv214078928:84 , srv231731022:84 , srv282949019:84 , srv290587561:84 , srv319381570:84 , srv327135737:84 , srv3378821:84 , srv338181852:84 , srv352986343:84 , srv370511549:84 , srv376723096:84 , srv461979919:84 , srv49410684:84 , srv526510828:84 , srv545477604:84 , srv62948544:84 , srv642407984:84 , srv642955976:84 , srv651375628:84 , srv660455129:84 , srv685214141:84 , srv710563956:84 , srv713490987:84 , srv718500047:84 , srv720903511:84 , srv780085392:84 , srv807619219:84 , srv835792369:84 , srv837116009:84 , srv869938933:84 , srv880475102:84 , srv890444659:84 , srv891197784:84 , srv921023408:84 , srv939309898:84 , srv946674891:84 , srv974873559:84 , srv975878035:84 , srv1036452691:85 , srv1241390495:85 , srv1596305554:85 , srv171674455:85 , srv387629307:85 , srv605181101:85 , srv1422678691:684 } 2023-05-10 20:53:37,050 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=301 (was 292) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=8988 (was 9843) 2023-05-10 20:53:37,059 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=301, ProcessCount=171, AvailableMemoryMB=8987 2023-05-10 20:53:37,060 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-10 20:53:37,060 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-10 20:53:37,073 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013814367:84 , srv1027290239:84 , srv1038209351:84 , srv1076493222:84 , srv1114460992:84 , srv1223333368:84 , srv1273698527:84 , srv1279485819:84 , srv1402343798:84 , srv1540995188:84 , srv1591520252:84 , srv1748921688:84 , srv1752996617:84 , srv184998542:84 , srv1860397602:84 , srv1965317814:84 , srv2092540948:84 , srv2094232906:84 , srv285069244:84 , srv314935420:84 , srv32195455:84 , srv363755972:84 , srv482454371:84 , srv599008644:84 , srv641943910:84 , srv891229255:84 , srv970006556:84 , srv997616794:84 , srv998027150:84 , srv1177444373:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-10 20:53:37,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-10 20:53:37,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1177444373=5, srv1402343798=9, srv1748921688=12, srv998027150=29, srv1860397602=15, srv1027290239=1, srv641943910=25, srv599008644=24, srv1223333368=6, srv997616794=28, srv1279485819=8, srv32195455=21, srv1273698527=7, srv1540995188=10, srv314935420=20, srv1038209351=2, srv1013814367=0, srv1965317814=16, srv363755972=22, srv891229255=26, srv1114460992=4, srv184998542=14, srv482454371=23, srv2094232906=18, srv285069244=19, srv1076493222=3, srv970006556=27, srv2092540948=17, srv1591520252=11, srv1752996617=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-10 20:53:37,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-10 20:53:37,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-10 20:53:37,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-10 20:53:37,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-10 20:53:37,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-10 20:53:37,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-10 20:53:37,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-10 20:53:37,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-10 20:53:37,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-10 20:53:37,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-10 20:53:37,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-10 20:53:37,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-10 20:53:37,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-10 20:53:37,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-10 20:53:37,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-10 20:53:37,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-10 20:53:37,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-10 20:53:37,115 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.133102957941792, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:53:37,116 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912612446338543, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.133102957941792, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-10 20:55:37,093 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6946419 different iterations. Found a solution that moves 1852 regions; Going from a computed imbalance of 0.9912612446338543 to a new imbalance of 4.322660390073086E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6859259259259259, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:55:37,099 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1013814367:90 , srv1027290239:90 , srv1038209351:90 , srv1076493222:90 , srv1114460992:90 , srv1177444373:90 , srv1223333368:90 , srv1273698527:90 , srv1279485819:90 , srv1402343798:90 , srv1540995188:90 , srv1591520252:90 , srv1748921688:90 , srv1752996617:90 , srv184998542:90 , srv1860397602:90 , srv1965317814:90 , srv2092540948:90 , srv2094232906:90 , srv285069244:90 , srv314935420:90 , srv32195455:90 , srv363755972:90 , srv482454371:90 , srv599008644:90 , srv641943910:90 , srv891229255:90 , srv970006556:90 , srv997616794:90 , srv998027150:90 } 2023-05-10 20:55:37,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-10 20:55:37,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1177444373=5, srv1402343798=9, srv1748921688=12, srv998027150=29, srv1860397602=15, srv1027290239=1, srv641943910=25, srv599008644=24, srv1223333368=6, srv997616794=28, srv1279485819=8, srv32195455=21, srv1273698527=7, srv1540995188=10, srv314935420=20, srv1038209351=2, srv1013814367=0, srv1965317814=16, srv363755972=22, srv891229255=26, srv1114460992=4, srv184998542=14, srv482454371=23, srv2094232906=18, srv285069244=19, srv1076493222=3, srv970006556=27, srv2092540948=17, srv1591520252=11, srv1752996617=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-10 20:55:37,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-10 20:55:37,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-10 20:55:37,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-10 20:55:37,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-10 20:55:37,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-10 20:55:37,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-10 20:55:37,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-10 20:55:37,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-10 20:55:37,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-10 20:55:37,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-10 20:55:37,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-10 20:55:37,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-10 20:55:37,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-10 20:55:37,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-10 20:55:37,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-10 20:55:37,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-10 20:55:37,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-10 20:55:37,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-10 20:55:37,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-10 20:55:37,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-10 20:55:37,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-10 20:55:37,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-10 20:55:37,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-10 20:55:37,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-10 20:55:37,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-10 20:55:37,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-10 20:55:37,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-10 20:55:37,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-10 20:55:37,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-10 20:55:37,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-10 20:55:37,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-10 20:55:37,121 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-10 20:55:37,121 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1013814367:90 , srv1027290239:90 , srv1038209351:90 , srv1076493222:90 , srv1114460992:90 , srv1177444373:90 , srv1223333368:90 , srv1273698527:90 , srv1279485819:90 , srv1402343798:90 , srv1540995188:90 , srv1591520252:90 , srv1748921688:90 , srv1752996617:90 , srv184998542:90 , srv1860397602:90 , srv1965317814:90 , srv2092540948:90 , srv2094232906:90 , srv285069244:90 , srv314935420:90 , srv32195455:90 , srv363755972:90 , srv482454371:90 , srv599008644:90 , srv641943910:90 , srv891229255:90 , srv970006556:90 , srv997616794:90 , srv998027150:90 } 2023-05-10 20:55:37,134 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=223 (was 301), ProcessCount=169 (was 171), AvailableMemoryMB=8715 (was 8987)