2023-07-12 00:59:04,379 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-07-12 00:59:04,640 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 00:59:04,734 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-12 00:59:04,790 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-12 00:59:04,797 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 00:59:04,807 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=128, ProcessCount=165, AvailableMemoryMB=5332 2023-07-12 00:59:04,809 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-12 00:59:04,809 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 00:59:04,922 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1028680067:84 , srv1112848362:84 , srv1128476891:84 , srv1161214044:84 , srv1171849433:84 , srv1188789760:84 , srv1189504730:84 , srv1191130095:84 , srv1217378893:84 , srv1229681373:84 , srv1252769205:84 , srv1270890509:84 , srv1275510716:84 , srv1277305958:84 , srv1352251056:84 , srv13566689:84 , srv1387954190:84 , srv1438823088:84 , srv1453857911:84 , srv1462303684:84 , srv1475051263:84 , srv1488661641:84 , srv1522547993:84 , srv1532871931:84 , srv1536843827:84 , srv1581179422:84 , srv1592926854:84 , srv1599552136:84 , srv1603733830:84 , srv1613967784:84 , srv1661790169:84 , srv1662488679:84 , srv1666325409:84 , srv169224686:84 , srv1745553230:84 , srv1753257240:84 , srv1776857269:84 , srv1796917781:84 , srv1812511084:84 , srv1835578196:84 , srv1850194324:84 , srv1854411206:84 , srv1889021612:84 , srv189123215:84 , srv1893001563:84 , srv1922315908:84 , srv1924164498:84 , srv1926901633:84 , srv1993553201:84 , srv1995475900:84 , srv2001257020:84 , srv200566131:84 , srv2015704450:84 , srv2019318771:84 , srv2061295186:84 , srv20957341:84 , srv2100808589:84 , srv2125100912:84 , srv2135705844:84 , srv217683867:84 , srv299904472:84 , srv308976278:84 , srv312977739:84 , srv353360733:84 , srv385845688:84 , srv39308909:84 , srv395463423:84 , srv399079715:84 , srv433252750:84 , srv454532895:84 , srv462858753:84 , srv464148733:84 , srv469290910:84 , srv487183632:84 , srv487603478:84 , srv491760072:84 , srv5052562:84 , srv536561491:84 , srv559331078:84 , srv579559099:84 , srv606813800:84 , srv629149303:84 , srv631793626:84 , srv633336827:84 , srv675690724:84 , srv688160627:84 , srv739428020:84 , srv773992137:84 , srv798527921:84 , srv81513021:84 , srv821981646:84 , srv824299483:84 , srv833850083:84 , srv834753883:84 , srv843513756:84 , srv860119694:84 , srv893685926:84 , srv896553314:84 , srv9388899:84 , srv1784559643:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-07-12 00:59:04,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-12 00:59:04,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1812511084=39, srv1462303684=19, srv1352251056=14, srv1889021612=43, srv13566689=15, srv773992137=88, srv1453857911=18, srv675690724=85, srv399079715=68, srv1993553201=49, srv559331078=79, srv9388899=99, srv1252769205=10, srv1753257240=35, srv1188789760=5, srv385845688=65, srv189123215=44, srv1666325409=32, srv834753883=94, srv1277305958=13, srv631793626=83, srv833850083=93, srv1835578196=40, srv896553314=98, srv20957341=56, srv462858753=71, srv1796917781=38, srv798527921=89, srv1745553230=34, srv1926901633=48, srv536561491=78, srv843513756=95, srv606813800=81, srv739428020=87, srv579559099=80, srv200566131=52, srv1488661641=21, srv1613967784=29, srv169224686=33, srv1532871931=23, srv39308909=66, srv1536843827=24, srv487183632=74, srv299904472=61, srv353360733=64, srv1028680067=0, srv1995475900=50, srv312977739=63, srv1924164498=47, srv2001257020=51, srv1599552136=27, srv1112848362=1, srv433252750=69, srv1475051263=20, srv487603478=75, srv1387954190=16, srv464148733=72, srv81513021=90, srv395463423=67, srv629149303=82, srv2100808589=57, srv2019318771=54, srv1275510716=12, srv893685926=97, srv5052562=77, srv688160627=86, srv1661790169=30, srv1161214044=3, srv1662488679=31, srv1229681373=9, srv1522547993=22, srv1128476891=2, srv308976278=62, srv824299483=92, srv1603733830=28, srv2125100912=58, srv1893001563=45, srv821981646=91, srv217683867=60, srv1922315908=46, srv1438823088=17, srv1270890509=11, srv454532895=70, srv2135705844=59, srv2015704450=53, srv1850194324=41, srv1171849433=4, srv491760072=76, srv1191130095=7, srv1189504730=6, srv1854411206=42, srv633336827=84, srv1784559643=37, srv1581179422=25, srv1592926854=26, srv1217378893=8, srv469290910=73, srv1776857269=36, srv2061295186=55, srv860119694=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-12 00:59:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:59:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-12 00:59:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-12 00:59:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-12 00:59:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-12 00:59:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-12 00:59:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-12 00:59:05,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-12 00:59:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-12 00:59:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-12 00:59:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-12 00:59:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-12 00:59:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-12 00:59:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-12 00:59:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-12 00:59:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-12 00:59:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-12 00:59:05,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-12 00:59:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-12 00:59:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-12 00:59:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-12 00:59:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-12 00:59:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-12 00:59:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-12 00:59:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-12 00:59:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-12 00:59:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-12 00:59:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-12 00:59:05,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-12 00:59:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-12 00:59:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-12 00:59:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-12 00:59:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-12 00:59:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-12 00:59:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-12 00:59:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-12 00:59:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-12 00:59:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-12 00:59:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-12 00:59:05,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-12 00:59:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-12 00:59:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-12 00:59:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-12 00:59:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-12 00:59:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-12 00:59:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-12 00:59:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-12 00:59:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-12 00:59:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-12 00:59:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-12 00:59:05,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-12 00:59:05,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-12 00:59:05,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-12 00:59:05,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-12 00:59:05,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-12 00:59:05,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-12 00:59:05,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-12 00:59:05,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-12 00:59:05,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-12 00:59:05,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-12 00:59:05,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-12 00:59:05,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-12 00:59:05,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-12 00:59:05,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-12 00:59:05,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-12 00:59:05,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-12 00:59:05,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-12 00:59:05,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-12 00:59:05,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-12 00:59:05,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-12 00:59:05,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-12 00:59:05,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-12 00:59:05,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-12 00:59:05,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-12 00:59:05,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-12 00:59:05,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-12 00:59:05,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-12 00:59:05,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-12 00:59:05,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-12 00:59:05,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-12 00:59:05,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-12 00:59:05,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-12 00:59:05,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-12 00:59:05,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-12 00:59:05,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-12 00:59:05,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-12 00:59:05,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-12 00:59:05,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-12 00:59:05,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-12 00:59:05,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-12 00:59:05,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-12 00:59:05,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-12 00:59:05,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-12 00:59:05,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-12 00:59:05,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-12 00:59:05,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-12 00:59:05,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-12 00:59:05,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-12 00:59:05,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-12 00:59:05,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-12 00:59:05,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-12 00:59:05,252 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-12 00:59:05,252 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-12 00:59:05,253 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-12 00:59:05,688 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 651 ms to try 5000 different iterations. Found a solution that moves 4680 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1802114052687626. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03958917347194236, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05504411046170846, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.52, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03958917347194236, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17475, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.249, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-12 00:59:05,698 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1275510716:80 , srv2019318771:82 , srv469290910:82 , srv491760072:82 , srv1522547993:83 , srv1581179422:83 , srv312977739:83 , srv1112848362:84 , srv1161214044:84 , srv1270890509:84 , srv1753257240:84 , srv1889021612:84 , srv2100808589:84 , srv454532895:84 , srv606813800:84 , srv631793626:84 , srv633336827:84 , srv81513021:84 , srv893685926:84 , srv896553314:84 , srv1191130095:85 , srv1387954190:85 , srv1462303684:85 , srv1592926854:85 , srv1599552136:85 , srv1603733830:85 , srv1662488679:85 , srv1776857269:85 , srv200566131:85 , srv217683867:85 , srv308976278:85 , srv395463423:85 , srv399079715:85 , srv462858753:85 , srv798527921:85 , srv1128476891:86 , srv1188789760:86 , srv1217378893:86 , srv13566689:86 , srv1666325409:86 , srv1745553230:86 , srv1796917781:86 , srv189123215:86 , srv1893001563:86 , srv1924164498:86 , srv1995475900:86 , srv2015704450:86 , srv20957341:86 , srv2135705844:86 , srv39308909:86 , srv433252750:86 , srv860119694:86 , srv1028680067:87 , srv1229681373:87 , srv1352251056:87 , srv1488661641:87 , srv1532871931:87 , srv1661790169:87 , srv1850194324:87 , srv1922315908:87 , srv1993553201:87 , srv2001257020:87 , srv2061295186:87 , srv487603478:87 , srv559331078:87 , srv629149303:87 , srv739428020:87 , srv821981646:87 , srv843513756:87 , srv1189504730:88 , srv1277305958:88 , srv1613967784:88 , srv1812511084:88 , srv1835578196:88 , srv1854411206:88 , srv1926901633:88 , srv299904472:88 , srv5052562:88 , srv536561491:88 , srv9388899:88 , srv1252769205:89 , srv1438823088:89 , srv1453857911:89 , srv1536843827:89 , srv487183632:89 , srv579559099:89 , srv688160627:89 , srv773992137:89 , srv834753883:89 , srv1171849433:90 , srv169224686:90 , srv2125100912:90 , srv385845688:90 , srv464148733:90 , srv675690724:90 , srv833850083:90 , srv1475051263:91 , srv824299483:91 , srv353360733:94 , srv1784559643:442 } 2023-07-12 00:59:05,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-12 00:59:05,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1812511084=39, srv1462303684=19, srv1352251056=14, srv1889021612=43, srv13566689=15, srv773992137=88, srv1453857911=18, srv675690724=85, srv399079715=68, srv1993553201=49, srv559331078=79, srv9388899=99, srv1252769205=10, srv1753257240=35, srv1188789760=5, srv385845688=65, srv189123215=44, srv1666325409=32, srv834753883=94, srv1277305958=13, srv631793626=83, srv833850083=93, srv1835578196=40, srv896553314=98, srv20957341=56, srv462858753=71, srv1796917781=38, srv798527921=89, srv1745553230=34, srv1926901633=48, srv536561491=78, srv843513756=95, srv606813800=81, srv739428020=87, srv579559099=80, srv200566131=52, srv1488661641=21, srv1613967784=29, srv169224686=33, srv1532871931=23, srv39308909=66, srv1536843827=24, srv487183632=74, srv299904472=61, srv353360733=64, srv1028680067=0, srv1995475900=50, srv312977739=63, srv1924164498=47, srv2001257020=51, srv1599552136=27, srv1112848362=1, srv433252750=69, srv1475051263=20, srv487603478=75, srv1387954190=16, srv464148733=72, srv81513021=90, srv395463423=67, srv629149303=82, srv2100808589=57, srv2019318771=54, srv1275510716=12, srv893685926=97, srv5052562=77, srv688160627=86, srv1661790169=30, srv1161214044=3, srv1662488679=31, srv1229681373=9, srv1522547993=22, srv1128476891=2, srv308976278=62, srv824299483=92, srv1603733830=28, srv2125100912=58, srv1893001563=45, srv821981646=91, srv217683867=60, srv1922315908=46, srv1438823088=17, srv1270890509=11, srv454532895=70, srv2135705844=59, srv2015704450=53, srv1850194324=41, srv1171849433=4, srv491760072=76, srv1191130095=7, srv1189504730=6, srv1854411206=42, srv633336827=84, srv1784559643=37, srv1581179422=25, srv1592926854=26, srv1217378893=8, srv469290910=73, srv1776857269=36, srv2061295186=55, srv860119694=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-12 00:59:05,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:59:05,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-12 00:59:05,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-12 00:59:05,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-12 00:59:05,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-12 00:59:05,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-12 00:59:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-12 00:59:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-12 00:59:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-12 00:59:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-12 00:59:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-12 00:59:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-12 00:59:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-12 00:59:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-12 00:59:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-12 00:59:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-12 00:59:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-12 00:59:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-12 00:59:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-12 00:59:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-12 00:59:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-12 00:59:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-12 00:59:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-12 00:59:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-12 00:59:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-12 00:59:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-12 00:59:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-12 00:59:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-12 00:59:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-12 00:59:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-12 00:59:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-12 00:59:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-12 00:59:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-12 00:59:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-12 00:59:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-12 00:59:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-12 00:59:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-12 00:59:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-12 00:59:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-12 00:59:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-12 00:59:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-12 00:59:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-12 00:59:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-12 00:59:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-12 00:59:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-12 00:59:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-12 00:59:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-12 00:59:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-12 00:59:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-12 00:59:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-12 00:59:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-12 00:59:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-12 00:59:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-12 00:59:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-12 00:59:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-12 00:59:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-12 00:59:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-12 00:59:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-12 00:59:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-12 00:59:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-12 00:59:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-12 00:59:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-12 00:59:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-12 00:59:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-12 00:59:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-12 00:59:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-12 00:59:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-12 00:59:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-12 00:59:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-12 00:59:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-12 00:59:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-12 00:59:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-12 00:59:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-12 00:59:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-12 00:59:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-12 00:59:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-12 00:59:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-12 00:59:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-12 00:59:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-12 00:59:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-12 00:59:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-12 00:59:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-12 00:59:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-12 00:59:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-12 00:59:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-12 00:59:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-12 00:59:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-12 00:59:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-12 00:59:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-12 00:59:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-12 00:59:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-12 00:59:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-12 00:59:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-12 00:59:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-12 00:59:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-12 00:59:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-12 00:59:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-12 00:59:05,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-12 00:59:05,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-12 00:59:05,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-12 00:59:05,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-12 00:59:05,859 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03958917347194236, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05504411046170846, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03958917347194236, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17475, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.249, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-12 00:59:05,860 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-12 00:59:05,860 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18017863520835406, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03958917347194236, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05504411046170846, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03958917347194236, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17475, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.249, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-12 00:59:06,050 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 294 ms to try 5000 different iterations. Found a solution that moves 2700 regions; Going from a computed imbalance of 0.18017863520835406 to a new imbalance of 0.014559577275739614. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021711872608003044); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03036995975742788, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.3, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021711872608003044); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-12 00:59:06,058 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv353360733:77 , srv824299483:79 , srv464148733:80 , srv1171849433:81 , srv1475051263:81 , srv2125100912:81 , srv675690724:81 , srv833850083:81 , srv1252769205:82 , srv1536843827:82 , srv1613967784:82 , srv385845688:82 , srv536561491:82 , srv773992137:82 , srv1277305958:83 , srv1453857911:83 , srv169224686:83 , srv1835578196:83 , srv1926901633:83 , srv688160627:83 , srv1028680067:84 , srv1229681373:84 , srv1438823088:84 , srv1661790169:84 , srv1850194324:84 , srv1854411206:84 , srv2001257020:84 , srv2015704450:84 , srv299904472:84 , srv487183632:84 , srv579559099:84 , srv739428020:84 , srv1488661641:85 , srv1592926854:85 , srv1603733830:85 , srv1666325409:85 , srv1796917781:85 , srv1812511084:85 , srv189123215:85 , srv1993553201:85 , srv39308909:85 , srv399079715:85 , srv5052562:85 , srv834753883:85 , srv1128476891:86 , srv1188789760:86 , srv1217378893:86 , srv1462303684:86 , srv1532871931:86 , srv1745553230:86 , srv1893001563:86 , srv1922315908:86 , srv1924164498:86 , srv200566131:86 , srv2061295186:86 , srv20957341:86 , srv487603478:86 , srv606813800:86 , srv629149303:86 , srv81513021:86 , srv843513756:86 , srv9388899:86 , srv1189504730:87 , srv1270890509:87 , srv1352251056:87 , srv13566689:87 , srv1662488679:87 , srv1753257240:87 , srv1776857269:87 , srv1889021612:87 , srv2135705844:87 , srv217683867:87 , srv312977739:87 , srv395463423:87 , srv433252750:87 , srv559331078:87 , srv631793626:87 , srv798527921:87 , srv1995475900:88 , srv2100808589:88 , srv308976278:88 , srv454532895:88 , srv633336827:88 , srv821981646:88 , srv860119694:88 , srv893685926:88 , srv1112848362:89 , srv1191130095:89 , srv1387954190:89 , srv1522547993:89 , srv1599552136:89 , srv462858753:89 , srv469290910:90 , srv1581179422:91 , srv2019318771:91 , srv491760072:91 , srv896553314:91 , srv1161214044:92 , srv1275510716:92 , srv1784559643:525 } 2023-07-12 00:59:06,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-12 00:59:06,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1812511084=39, srv1462303684=19, srv1352251056=14, srv1889021612=43, srv13566689=15, srv773992137=88, srv1453857911=18, srv675690724=85, srv399079715=68, srv1993553201=49, srv559331078=79, srv9388899=99, srv1252769205=10, srv1753257240=35, srv1188789760=5, srv385845688=65, srv189123215=44, srv1666325409=32, srv834753883=94, srv1277305958=13, srv631793626=83, srv833850083=93, srv1835578196=40, srv896553314=98, srv20957341=56, srv462858753=71, srv1796917781=38, srv798527921=89, srv1745553230=34, srv1926901633=48, srv536561491=78, srv843513756=95, srv606813800=81, srv739428020=87, srv579559099=80, srv200566131=52, srv1488661641=21, srv1613967784=29, srv169224686=33, srv1532871931=23, srv39308909=66, srv1536843827=24, srv487183632=74, srv299904472=61, srv353360733=64, srv1028680067=0, srv1995475900=50, srv312977739=63, srv1924164498=47, srv2001257020=51, srv1599552136=27, srv1112848362=1, srv433252750=69, srv1475051263=20, srv487603478=75, srv1387954190=16, srv464148733=72, srv81513021=90, srv395463423=67, srv629149303=82, srv2100808589=57, srv2019318771=54, srv1275510716=12, srv893685926=97, srv5052562=77, srv688160627=86, srv1661790169=30, srv1161214044=3, srv1662488679=31, srv1229681373=9, srv1522547993=22, srv1128476891=2, srv308976278=62, srv824299483=92, srv1603733830=28, srv2125100912=58, srv1893001563=45, srv821981646=91, srv217683867=60, srv1922315908=46, srv1438823088=17, srv1270890509=11, srv454532895=70, srv2135705844=59, srv2015704450=53, srv1850194324=41, srv1171849433=4, srv491760072=76, srv1191130095=7, srv1189504730=6, srv1854411206=42, srv633336827=84, srv1784559643=37, srv1581179422=25, srv1592926854=26, srv1217378893=8, srv469290910=73, srv1776857269=36, srv2061295186=55, srv860119694=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-12 00:59:06,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:59:06,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-12 00:59:06,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-12 00:59:06,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-12 00:59:06,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-12 00:59:06,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-12 00:59:06,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-12 00:59:06,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-12 00:59:06,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-12 00:59:06,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-12 00:59:06,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-12 00:59:06,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-12 00:59:06,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-12 00:59:06,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-12 00:59:06,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-12 00:59:06,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-12 00:59:06,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-12 00:59:06,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-12 00:59:06,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-12 00:59:06,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-12 00:59:06,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-12 00:59:06,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-12 00:59:06,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-12 00:59:06,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-12 00:59:06,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-12 00:59:06,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-12 00:59:06,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-12 00:59:06,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-12 00:59:06,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-12 00:59:06,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-12 00:59:06,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-12 00:59:06,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-12 00:59:06,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-12 00:59:06,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-12 00:59:06,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-12 00:59:06,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-12 00:59:06,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-12 00:59:06,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-12 00:59:06,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-12 00:59:06,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-12 00:59:06,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-12 00:59:06,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-12 00:59:06,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-12 00:59:06,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-12 00:59:06,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-12 00:59:06,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-12 00:59:06,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-12 00:59:06,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-12 00:59:06,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-12 00:59:06,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-12 00:59:06,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-12 00:59:06,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-12 00:59:06,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-12 00:59:06,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-12 00:59:06,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-12 00:59:06,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-12 00:59:06,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-12 00:59:06,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-12 00:59:06,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-12 00:59:06,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-12 00:59:06,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-12 00:59:06,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-12 00:59:06,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-12 00:59:06,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-12 00:59:06,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-12 00:59:06,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-12 00:59:06,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-12 00:59:06,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-12 00:59:06,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-12 00:59:06,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-12 00:59:06,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-12 00:59:06,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-12 00:59:06,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-12 00:59:06,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-12 00:59:06,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-12 00:59:06,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-12 00:59:06,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-12 00:59:06,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-12 00:59:06,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-12 00:59:06,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-12 00:59:06,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-12 00:59:06,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-12 00:59:06,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-12 00:59:06,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-12 00:59:06,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-12 00:59:06,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-12 00:59:06,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-12 00:59:06,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-12 00:59:06,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-12 00:59:06,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-12 00:59:06,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-12 00:59:06,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-12 00:59:06,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-12 00:59:06,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-12 00:59:06,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-12 00:59:06,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-12 00:59:06,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-12 00:59:06,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-12 00:59:06,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-12 00:59:06,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-12 00:59:06,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-12 00:59:06,183 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021711872608003044); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03036995975742788, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021711872608003044); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-12 00:59:06,184 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-12 00:59:06,184 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01454067147165776, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021711872608003044); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03036995975742788, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021711872608003044); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-12 00:59:06,342 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 231 ms to try 5000 different iterations. Found a solution that moves 1023 regions; Going from a computed imbalance of 0.01454067147165776 to a new imbalance of 8.061147909941184E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054295474061417486); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005014009553935699); MoveCostFunction : (multiplier=7.0, imbalance=0.11366666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054295474061417486); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-12 00:59:06,345 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1161214044:81 , srv1189504730:82 , srv896553314:82 , srv1191130095:83 , srv1352251056:83 , srv1581179422:83 , srv1599552136:83 , srv1995475900:83 , srv821981646:83 , srv834753883:83 , srv1112848362:84 , srv13566689:84 , srv1387954190:84 , srv1438823088:84 , srv1532871931:84 , srv169224686:84 , srv2019318771:84 , srv2135705844:84 , srv308976278:84 , srv491760072:84 , srv559331078:84 , srv629149303:84 , srv860119694:84 , srv9388899:84 , srv1188789760:85 , srv1217378893:85 , srv1275510716:85 , srv1453857911:85 , srv1475051263:85 , srv1488661641:85 , srv1662488679:85 , srv1812511084:85 , srv1854411206:85 , srv1922315908:85 , srv1924164498:85 , srv2061295186:85 , srv20957341:85 , srv299904472:85 , srv433252750:85 , srv462858753:85 , srv487183632:85 , srv5052562:85 , srv579559099:85 , srv633336827:85 , srv798527921:85 , srv843513756:85 , srv893685926:85 , srv1028680067:86 , srv1128476891:86 , srv1171849433:86 , srv1229681373:86 , srv1270890509:86 , srv1522547993:86 , srv1536843827:86 , srv1666325409:86 , srv1745553230:86 , srv1776857269:86 , srv1796917781:86 , srv1889021612:86 , srv189123215:86 , srv1893001563:86 , srv1926901633:86 , srv1993553201:86 , srv2001257020:86 , srv200566131:86 , srv2100808589:86 , srv217683867:86 , srv385845688:86 , srv395463423:86 , srv454532895:86 , srv469290910:86 , srv487603478:86 , srv688160627:86 , srv1252769205:87 , srv1277305958:87 , srv1462303684:87 , srv1603733830:87 , srv1613967784:87 , srv1661790169:87 , srv1753257240:87 , srv1835578196:87 , srv1850194324:87 , srv2015704450:87 , srv2125100912:87 , srv312977739:87 , srv39308909:87 , srv399079715:87 , srv464148733:87 , srv536561491:87 , srv675690724:87 , srv739428020:87 , srv773992137:87 , srv81513021:87 , srv833850083:87 , srv1592926854:88 , srv353360733:88 , srv606813800:88 , srv824299483:88 , srv631793626:89 , srv1784559643:539 } 2023-07-12 00:59:06,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-12 00:59:06,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1812511084=39, srv1462303684=19, srv1352251056=14, srv1889021612=43, srv13566689=15, srv773992137=88, srv1453857911=18, srv675690724=85, srv399079715=68, srv1993553201=49, srv559331078=79, srv9388899=99, srv1252769205=10, srv1753257240=35, srv1188789760=5, srv385845688=65, srv189123215=44, srv1666325409=32, srv834753883=94, srv1277305958=13, srv631793626=83, srv833850083=93, srv1835578196=40, srv896553314=98, srv20957341=56, srv462858753=71, srv1796917781=38, srv798527921=89, srv1745553230=34, srv1926901633=48, srv536561491=78, srv843513756=95, srv606813800=81, srv739428020=87, srv579559099=80, srv200566131=52, srv1488661641=21, srv1613967784=29, srv169224686=33, srv1532871931=23, srv39308909=66, srv1536843827=24, srv487183632=74, srv299904472=61, srv353360733=64, srv1028680067=0, srv1995475900=50, srv312977739=63, srv1924164498=47, srv2001257020=51, srv1599552136=27, srv1112848362=1, srv433252750=69, srv1475051263=20, srv487603478=75, srv1387954190=16, srv464148733=72, srv81513021=90, srv395463423=67, srv629149303=82, srv2100808589=57, srv2019318771=54, srv1275510716=12, srv893685926=97, srv5052562=77, srv688160627=86, srv1661790169=30, srv1161214044=3, srv1662488679=31, srv1229681373=9, srv1522547993=22, srv1128476891=2, srv308976278=62, srv824299483=92, srv1603733830=28, srv2125100912=58, srv1893001563=45, srv821981646=91, srv217683867=60, srv1922315908=46, srv1438823088=17, srv1270890509=11, srv454532895=70, srv2135705844=59, srv2015704450=53, srv1850194324=41, srv1171849433=4, srv491760072=76, srv1191130095=7, srv1189504730=6, srv1854411206=42, srv633336827=84, srv1784559643=37, srv1581179422=25, srv1592926854=26, srv1217378893=8, srv469290910=73, srv1776857269=36, srv2061295186=55, srv860119694=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-12 00:59:06,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:59:06,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-12 00:59:06,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-12 00:59:06,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-12 00:59:06,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-12 00:59:06,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-12 00:59:06,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-12 00:59:06,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-12 00:59:06,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-12 00:59:06,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-12 00:59:06,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-12 00:59:06,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-12 00:59:06,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-12 00:59:06,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-12 00:59:06,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-12 00:59:06,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-12 00:59:06,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-12 00:59:06,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-12 00:59:06,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-12 00:59:06,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-12 00:59:06,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-12 00:59:06,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-12 00:59:06,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-12 00:59:06,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-12 00:59:06,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-12 00:59:06,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-12 00:59:06,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-12 00:59:06,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-12 00:59:06,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-12 00:59:06,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-12 00:59:06,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-12 00:59:06,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-12 00:59:06,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-12 00:59:06,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-12 00:59:06,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-12 00:59:06,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-12 00:59:06,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-12 00:59:06,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-12 00:59:06,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-12 00:59:06,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-12 00:59:06,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-12 00:59:06,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-12 00:59:06,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-12 00:59:06,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-12 00:59:06,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-12 00:59:06,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-12 00:59:06,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-12 00:59:06,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-12 00:59:06,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-12 00:59:06,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-12 00:59:06,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-12 00:59:06,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-12 00:59:06,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-12 00:59:06,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-12 00:59:06,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-12 00:59:06,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-12 00:59:06,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-12 00:59:06,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-12 00:59:06,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-12 00:59:06,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-12 00:59:06,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-12 00:59:06,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-12 00:59:06,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-12 00:59:06,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-12 00:59:06,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-12 00:59:06,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-12 00:59:06,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-12 00:59:06,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-12 00:59:06,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-12 00:59:06,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-12 00:59:06,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-12 00:59:06,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-12 00:59:06,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-12 00:59:06,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-12 00:59:06,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-12 00:59:06,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-12 00:59:06,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-12 00:59:06,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-12 00:59:06,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-12 00:59:06,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-12 00:59:06,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-12 00:59:06,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-12 00:59:06,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-12 00:59:06,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-12 00:59:06,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-12 00:59:06,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-12 00:59:06,442 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054295474061417486); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005014009553935699); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054295474061417486); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-12 00:59:06,442 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-12 00:59:06,442 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.98951591891994E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054295474061417486); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005014009553935699); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054295474061417486); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-12 00:59:06,660 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 286 ms to try 5000 different iterations. Found a solution that moves 252 regions; Going from a computed imbalance of 7.98951591891994E-4 to a new imbalance of 2.04664044481532E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.028, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-12 00:59:06,661 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1252769205:83 , srv1745553230:83 , srv353360733:83 , srv462858753:83 , srv5052562:83 , srv631793626:83 , srv1128476891:84 , srv1277305958:84 , srv1387954190:84 , srv1438823088:84 , srv1462303684:84 , srv1522547993:84 , srv1592926854:84 , srv1661790169:84 , srv169224686:84 , srv1753257240:84 , srv1776857269:84 , srv1812511084:84 , srv1835578196:84 , srv1850194324:84 , srv1893001563:84 , srv1922315908:84 , srv1993553201:84 , srv1995475900:84 , srv2061295186:84 , srv2100808589:84 , srv2125100912:84 , srv217683867:84 , srv385845688:84 , srv39308909:84 , srv395463423:84 , srv433252750:84 , srv454532895:84 , srv464148733:84 , srv469290910:84 , srv487183632:84 , srv487603478:84 , srv559331078:84 , srv579559099:84 , srv606813800:84 , srv675690724:84 , srv688160627:84 , srv739428020:84 , srv773992137:84 , srv821981646:84 , srv824299483:84 , srv833850083:84 , srv843513756:84 , srv860119694:84 , srv893685926:84 , srv9388899:84 , srv1028680067:85 , srv1112848362:85 , srv1171849433:85 , srv1188789760:85 , srv1191130095:85 , srv1217378893:85 , srv1229681373:85 , srv1270890509:85 , srv1275510716:85 , srv1352251056:85 , srv13566689:85 , srv1453857911:85 , srv1488661641:85 , srv1532871931:85 , srv1536843827:85 , srv1581179422:85 , srv1599552136:85 , srv1613967784:85 , srv1662488679:85 , srv1666325409:85 , srv1796917781:85 , srv1854411206:85 , srv1889021612:85 , srv189123215:85 , srv1924164498:85 , srv1926901633:85 , srv200566131:85 , srv2015704450:85 , srv2019318771:85 , srv20957341:85 , srv2135705844:85 , srv299904472:85 , srv308976278:85 , srv312977739:85 , srv399079715:85 , srv491760072:85 , srv536561491:85 , srv629149303:85 , srv633336827:85 , srv798527921:85 , srv81513021:85 , srv834753883:85 , srv896553314:85 , srv1161214044:86 , srv1189504730:86 , srv1475051263:86 , srv1603733830:86 , srv2001257020:86 , srv1784559643:637 } 2023-07-12 00:59:06,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-12 00:59:06,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1812511084=39, srv1462303684=19, srv1352251056=14, srv1889021612=43, srv13566689=15, srv773992137=88, srv1453857911=18, srv675690724=85, srv399079715=68, srv1993553201=49, srv559331078=79, srv9388899=99, srv1252769205=10, srv1753257240=35, srv1188789760=5, srv385845688=65, srv189123215=44, srv1666325409=32, srv834753883=94, srv1277305958=13, srv631793626=83, srv833850083=93, srv1835578196=40, srv896553314=98, srv20957341=56, srv462858753=71, srv1796917781=38, srv798527921=89, srv1745553230=34, srv1926901633=48, srv536561491=78, srv843513756=95, srv606813800=81, srv739428020=87, srv579559099=80, srv200566131=52, srv1488661641=21, srv1613967784=29, srv169224686=33, srv1532871931=23, srv39308909=66, srv1536843827=24, srv487183632=74, srv299904472=61, srv353360733=64, srv1028680067=0, srv1995475900=50, srv312977739=63, srv1924164498=47, srv2001257020=51, srv1599552136=27, srv1112848362=1, srv433252750=69, srv1475051263=20, srv487603478=75, srv1387954190=16, srv464148733=72, srv81513021=90, srv395463423=67, srv629149303=82, srv2100808589=57, srv2019318771=54, srv1275510716=12, srv893685926=97, srv5052562=77, srv688160627=86, srv1661790169=30, srv1161214044=3, srv1662488679=31, srv1229681373=9, srv1522547993=22, srv1128476891=2, srv308976278=62, srv824299483=92, srv1603733830=28, srv2125100912=58, srv1893001563=45, srv821981646=91, srv217683867=60, srv1922315908=46, srv1438823088=17, srv1270890509=11, srv454532895=70, srv2135705844=59, srv2015704450=53, srv1850194324=41, srv1171849433=4, srv491760072=76, srv1191130095=7, srv1189504730=6, srv1854411206=42, srv633336827=84, srv1784559643=37, srv1581179422=25, srv1592926854=26, srv1217378893=8, srv469290910=73, srv1776857269=36, srv2061295186=55, srv860119694=96} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-12 00:59:06,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:59:06,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-12 00:59:06,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-12 00:59:06,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-12 00:59:06,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-12 00:59:06,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-12 00:59:06,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-12 00:59:06,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-12 00:59:06,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-12 00:59:06,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-12 00:59:06,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-12 00:59:06,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-12 00:59:06,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-12 00:59:06,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-12 00:59:06,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-12 00:59:06,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-12 00:59:06,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-12 00:59:06,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-12 00:59:06,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-12 00:59:06,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-12 00:59:06,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-12 00:59:06,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-12 00:59:06,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-12 00:59:06,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-12 00:59:06,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-12 00:59:06,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-12 00:59:06,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-12 00:59:06,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-12 00:59:06,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-12 00:59:06,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-12 00:59:06,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-12 00:59:06,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-12 00:59:06,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-12 00:59:06,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-12 00:59:06,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-12 00:59:06,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-12 00:59:06,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-12 00:59:06,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-12 00:59:06,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-12 00:59:06,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-12 00:59:06,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-12 00:59:06,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-12 00:59:06,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-12 00:59:06,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-12 00:59:06,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-12 00:59:06,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-12 00:59:06,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-12 00:59:06,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-12 00:59:06,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-12 00:59:06,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-12 00:59:06,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-12 00:59:06,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-12 00:59:06,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-12 00:59:06,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-12 00:59:06,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-12 00:59:06,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-12 00:59:06,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-12 00:59:06,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-12 00:59:06,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-12 00:59:06,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-12 00:59:06,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-12 00:59:06,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-12 00:59:06,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-12 00:59:06,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-12 00:59:06,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-12 00:59:06,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-12 00:59:06,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-12 00:59:06,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-12 00:59:06,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-12 00:59:06,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-12 00:59:06,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-12 00:59:06,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-12 00:59:06,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-12 00:59:06,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-12 00:59:06,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-12 00:59:06,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-12 00:59:06,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-12 00:59:06,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-12 00:59:06,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-12 00:59:06,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-12 00:59:06,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-12 00:59:06,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-12 00:59:06,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-12 00:59:06,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-12 00:59:06,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-12 00:59:06,763 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8701862733846913E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-12 00:59:06,764 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1252769205:83 , srv1745553230:83 , srv353360733:83 , srv462858753:83 , srv5052562:83 , srv631793626:83 , srv1128476891:84 , srv1277305958:84 , srv1387954190:84 , srv1438823088:84 , srv1462303684:84 , srv1522547993:84 , srv1592926854:84 , srv1661790169:84 , srv169224686:84 , srv1753257240:84 , srv1776857269:84 , srv1812511084:84 , srv1835578196:84 , srv1850194324:84 , srv1893001563:84 , srv1922315908:84 , srv1993553201:84 , srv1995475900:84 , srv2061295186:84 , srv2100808589:84 , srv2125100912:84 , srv217683867:84 , srv385845688:84 , srv39308909:84 , srv395463423:84 , srv433252750:84 , srv454532895:84 , srv464148733:84 , srv469290910:84 , srv487183632:84 , srv487603478:84 , srv559331078:84 , srv579559099:84 , srv606813800:84 , srv675690724:84 , srv688160627:84 , srv739428020:84 , srv773992137:84 , srv821981646:84 , srv824299483:84 , srv833850083:84 , srv843513756:84 , srv860119694:84 , srv893685926:84 , srv9388899:84 , srv1028680067:85 , srv1112848362:85 , srv1171849433:85 , srv1188789760:85 , srv1191130095:85 , srv1217378893:85 , srv1229681373:85 , srv1270890509:85 , srv1275510716:85 , srv1352251056:85 , srv13566689:85 , srv1453857911:85 , srv1488661641:85 , srv1532871931:85 , srv1536843827:85 , srv1581179422:85 , srv1599552136:85 , srv1613967784:85 , srv1662488679:85 , srv1666325409:85 , srv1796917781:85 , srv1854411206:85 , srv1889021612:85 , srv189123215:85 , srv1924164498:85 , srv1926901633:85 , srv200566131:85 , srv2015704450:85 , srv2019318771:85 , srv20957341:85 , srv2135705844:85 , srv299904472:85 , srv308976278:85 , srv312977739:85 , srv399079715:85 , srv491760072:85 , srv536561491:85 , srv629149303:85 , srv633336827:85 , srv798527921:85 , srv81513021:85 , srv834753883:85 , srv896553314:85 , srv1161214044:86 , srv1189504730:86 , srv1475051263:86 , srv1603733830:86 , srv2001257020:86 , srv1784559643:637 } 2023-07-12 00:59:06,783 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=128 (was 128), ProcessCount=165 (was 165), AvailableMemoryMB=4843 (was 5332) 2023-07-12 00:59:06,790 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=128, ProcessCount=165, AvailableMemoryMB=4842 2023-07-12 00:59:06,791 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-12 00:59:06,791 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-07-12 00:59:06,800 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1037331931:84 , srv1054709853:84 , srv1115676252:84 , srv1168073534:84 , srv1230207843:84 , srv1276157794:84 , srv1292459238:84 , srv1364949012:84 , srv1374827068:84 , srv141446151:84 , srv1464567747:84 , srv1599262184:84 , srv1665622230:84 , srv1675418990:84 , srv1890886916:84 , srv1947418661:84 , srv1958031313:84 , srv2055160380:84 , srv2107884060:84 , srv336294672:84 , srv522655353:84 , srv557403218:84 , srv569059638:84 , srv577834139:84 , srv619546863:84 , srv626262002:84 , srv636760681:84 , srv658822762:84 , srv938247361:84 , srv647506031:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-07-12 00:59:06,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-12 00:59:06,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1464567747=10, srv647506031=27, srv1115676252=2, srv1054709853=1, srv2107884060=18, srv1168073534=3, srv619546863=24, srv1292459238=6, srv522655353=20, srv938247361=29, srv1599262184=11, srv1037331931=0, srv636760681=26, srv1890886916=14, srv1374827068=8, srv577834139=23, srv658822762=28, srv1947418661=15, srv2055160380=17, srv569059638=22, srv1665622230=12, srv557403218=21, srv1230207843=4, srv141446151=9, srv1276157794=5, srv336294672=19, srv1364949012=7, srv626262002=25, srv1675418990=13, srv1958031313=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-12 00:59:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 00:59:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-12 00:59:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-12 00:59:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-12 00:59:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-12 00:59:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-12 00:59:06,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-12 00:59:06,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-12 00:59:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-12 00:59:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-12 00:59:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-12 00:59:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-12 00:59:06,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-12 00:59:06,826 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.13623908965307, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-12 00:59:06,826 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912622328187161, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.13623908965307, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-07-12 01:01:06,815 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9911054 different iterations. Found a solution that moves 1835 regions; Going from a computed imbalance of 0.9912622328187161 to a new imbalance of 4.2829815420000605E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6796296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-12 01:01:06,822 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1037331931:90 , srv1054709853:90 , srv1115676252:90 , srv1168073534:90 , srv1230207843:90 , srv1276157794:90 , srv1292459238:90 , srv1364949012:90 , srv1374827068:90 , srv141446151:90 , srv1464567747:90 , srv1599262184:90 , srv1665622230:90 , srv1675418990:90 , srv1890886916:90 , srv1947418661:90 , srv1958031313:90 , srv2055160380:90 , srv2107884060:90 , srv336294672:90 , srv522655353:90 , srv557403218:90 , srv569059638:90 , srv577834139:90 , srv619546863:90 , srv626262002:90 , srv636760681:90 , srv647506031:90 , srv658822762:90 , srv938247361:90 } 2023-07-12 01:01:06,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-12 01:01:06,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1464567747=10, srv647506031=27, srv1115676252=2, srv1054709853=1, srv2107884060=18, srv1168073534=3, srv619546863=24, srv1292459238=6, srv522655353=20, srv938247361=29, srv1599262184=11, srv1037331931=0, srv636760681=26, srv1890886916=14, srv1374827068=8, srv577834139=23, srv658822762=28, srv1947418661=15, srv2055160380=17, srv569059638=22, srv1665622230=12, srv557403218=21, srv1230207843=4, srv141446151=9, srv1276157794=5, srv336294672=19, srv1364949012=7, srv626262002=25, srv1675418990=13, srv1958031313=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-12 01:01:06,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 01:01:06,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-12 01:01:06,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-12 01:01:06,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-12 01:01:06,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-12 01:01:06,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-12 01:01:06,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-12 01:01:06,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-12 01:01:06,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-12 01:01:06,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-12 01:01:06,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-12 01:01:06,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-12 01:01:06,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-12 01:01:06,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-12 01:01:06,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-12 01:01:06,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-12 01:01:06,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-12 01:01:06,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-12 01:01:06,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-12 01:01:06,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-12 01:01:06,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-12 01:01:06,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-12 01:01:06,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-12 01:01:06,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-12 01:01:06,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-12 01:01:06,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-12 01:01:06,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-12 01:01:06,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-12 01:01:06,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-12 01:01:06,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-12 01:01:06,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-12 01:01:06,840 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-12 01:01:06,840 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1037331931:90 , srv1054709853:90 , srv1115676252:90 , srv1168073534:90 , srv1230207843:90 , srv1276157794:90 , srv1292459238:90 , srv1364949012:90 , srv1374827068:90 , srv141446151:90 , srv1464567747:90 , srv1599262184:90 , srv1665622230:90 , srv1675418990:90 , srv1890886916:90 , srv1947418661:90 , srv1958031313:90 , srv2055160380:90 , srv2107884060:90 , srv336294672:90 , srv522655353:90 , srv557403218:90 , srv569059638:90 , srv577834139:90 , srv619546863:90 , srv626262002:90 , srv636760681:90 , srv647506031:90 , srv658822762:90 , srv938247361:90 } 2023-07-12 01:01:06,851 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=129 (was 128) - SystemLoadAverage LEAK? -, ProcessCount=165 (was 165), AvailableMemoryMB=4607 (was 4842)