2023-11-11 18:55:10,329 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-11-11 18:55:10,611 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-11-11 18:55:10,714 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-11-11 18:55:10,774 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-11 18:55:10,781 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-11-11 18:55:10,794 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=107, ProcessCount=167, AvailableMemoryMB=1615 2023-11-11 18:55:10,795 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-11 18:55:10,796 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-11-11 18:55:10,906 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1002245603:84 , srv1002956439:84 , srv1018746635:84 , srv1019184435:84 , srv102930737:84 , srv1035344488:84 , srv1049019563:84 , srv1055809399:84 , srv1060638226:84 , srv1125573121:84 , srv1134507230:84 , srv1149605709:84 , srv1182489063:84 , srv1186074813:84 , srv1265740510:84 , srv1275548031:84 , srv1275718627:84 , srv1276223047:84 , srv1304344444:84 , srv1340098439:84 , srv1412812751:84 , srv1416832389:84 , srv1435078148:84 , srv1437796997:84 , srv1480007736:84 , srv1556066751:84 , srv1558322610:84 , srv1600449813:84 , srv1603025151:84 , srv16081774:84 , srv1652092276:84 , srv1655603829:84 , srv1692914487:84 , srv1696670394:84 , srv1749831202:84 , srv1816622158:84 , srv1829366367:84 , srv1848478899:84 , srv1854626423:84 , srv1882434060:84 , srv1923829248:84 , srv1924243033:84 , srv1943776393:84 , srv1998440641:84 , srv2000826476:84 , srv2022150065:84 , srv2034890388:84 , srv203534509:84 , srv207071996:84 , srv2084222789:84 , srv2106613087:84 , srv2108926119:84 , srv222764382:84 , srv245037011:84 , srv274768748:84 , srv27928829:84 , srv293222208:84 , srv302716415:84 , srv306881934:84 , srv321342589:84 , srv358639470:84 , srv385173313:84 , srv403864605:84 , srv405616989:84 , srv413664870:84 , srv42885191:84 , srv430674973:84 , srv436905221:84 , srv447746112:84 , srv451973437:84 , srv466059494:84 , srv498874258:84 , srv509973775:84 , srv564159602:84 , srv588148747:84 , srv603297809:84 , srv628752794:84 , srv656655689:84 , srv664595304:84 , srv716816614:84 , srv717348547:84 , srv747764772:84 , srv750414013:84 , srv762911317:84 , srv765499841:84 , srv772055113:84 , srv780052633:84 , srv790361937:84 , srv791491322:84 , srv833950709:84 , srv836328591:84 , srv881791326:84 , srv882847609:84 , srv893404711:84 , srv9226263:84 , srv937283431:84 , srv978855970:84 , srv980493794:84 , srv997256211:84 , srv1591563966:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-11-11 18:55:10,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 18:55:10,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1437796997=23, srv2084222789=50, srv762911317=84, srv245037011=54, srv1412812751=20, srv1749831202=35, srv1998440641=44, srv2034890388=47, srv451973437=70, srv1002956439=1, srv717348547=81, srv1655603829=32, srv302716415=58, srv1018746635=2, srv978855970=97, srv1854626423=39, srv747764772=82, srv1304344444=18, srv293222208=57, srv207071996=49, srv1276223047=17, srv836328591=91, srv791491322=89, srv1692914487=33, srv1829366367=37, srv1882434060=40, srv413664870=65, srv1416832389=21, srv509973775=73, srv1924243033=42, srv1943776393=43, srv16081774=30, srv1035344488=5, srv588148747=75, srv1848478899=38, srv1186074813=13, srv1019184435=3, srv1275548031=15, srv1591563966=27, srv1149605709=11, srv1055809399=7, srv1060638226=8, srv1558322610=26, srv1652092276=31, srv980493794=98, srv498874258=72, srv772055113=86, srv882847609=93, srv203534509=48, srv1049019563=6, srv1696670394=34, srv306881934=59, srv102930737=4, srv750414013=83, srv405616989=64, srv1556066751=25, srv833950709=90, srv436905221=68, srv1125573121=9, srv321342589=60, srv1600449813=28, srv430674973=67, srv1816622158=36, srv447746112=69, srv274768748=55, srv2022150065=46, srv466059494=71, srv2108926119=52, srv628752794=77, srv1002245603=0, srv42885191=66, srv1182489063=12, srv937283431=96, srv1923829248=41, srv603297809=76, srv385173313=62, srv222764382=53, srv780052633=87, srv564159602=74, srv27928829=56, srv1134507230=10, srv1275718627=16, srv765499841=85, srv2000826476=45, srv716816614=80, srv358639470=61, srv1265740510=14, srv2106613087=51, srv1340098439=19, srv1603025151=29, srv1435078148=22, srv656655689=78, srv893404711=94, srv9226263=95, srv997256211=99, srv403864605=63, srv664595304=79, srv881791326=92, srv790361937=88, srv1480007736=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 18:55:10,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 18:55:10,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 18:55:10,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 18:55:10,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 18:55:10,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 18:55:10,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 18:55:10,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 18:55:10,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 18:55:10,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 18:55:10,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 18:55:10,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 18:55:10,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 18:55:10,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 18:55:10,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 18:55:10,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 18:55:10,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 18:55:10,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 18:55:10,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 18:55:10,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 18:55:10,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 18:55:10,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 18:55:10,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 18:55:10,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 18:55:10,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 18:55:10,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 18:55:10,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 18:55:10,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 18:55:10,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 18:55:10,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 18:55:10,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 18:55:10,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-11 18:55:10,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-11 18:55:10,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-11 18:55:10,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-11 18:55:10,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-11 18:55:10,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-11 18:55:10,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-11 18:55:10,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-11 18:55:10,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-11 18:55:10,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-11 18:55:10,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-11 18:55:10,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-11 18:55:10,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-11 18:55:10,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-11 18:55:10,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-11 18:55:10,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-11 18:55:10,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-11 18:55:10,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-11 18:55:10,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-11 18:55:10,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-11 18:55:10,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-11 18:55:10,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-11 18:55:10,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-11 18:55:10,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-11 18:55:10,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-11 18:55:10,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-11 18:55:10,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-11 18:55:10,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-11 18:55:10,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-11 18:55:10,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-11 18:55:10,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-11 18:55:10,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-11 18:55:10,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-11 18:55:10,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-11 18:55:10,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-11 18:55:10,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-11 18:55:10,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-11 18:55:10,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-11 18:55:10,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-11 18:55:10,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-11 18:55:10,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-11 18:55:10,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-11 18:55:10,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-11 18:55:10,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-11 18:55:10,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-11 18:55:10,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-11 18:55:10,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-11 18:55:10,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-11 18:55:10,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-11 18:55:10,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-11 18:55:10,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-11 18:55:10,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-11 18:55:10,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-11 18:55:10,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-11 18:55:10,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-11 18:55:10,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-11 18:55:10,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-11 18:55:10,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-11 18:55:10,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-11 18:55:10,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-11 18:55:10,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-11 18:55:10,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-11 18:55:10,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-11 18:55:10,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-11 18:55:10,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-11 18:55:10,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-11 18:55:10,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-11 18:55:10,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-11 18:55:10,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-11 18:55:10,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-11 18:55:10,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-11 18:55:11,161 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:55:11,161 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-11 18:55:11,162 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-11 18:55:11,522 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 500 ms to try 5000 different iterations. Found a solution that moves 4703 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17779964598980064. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04071777742802447, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0513520889257393, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5225555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04071777742802447, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17275, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24233333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:55:11,541 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1924243033:81 , srv1018746635:83 , srv1275718627:83 , srv1652092276:83 , srv2108926119:83 , srv451973437:83 , srv466059494:83 , srv509973775:83 , srv750414013:83 , srv772055113:83 , srv833950709:83 , srv102930737:84 , srv1265740510:84 , srv1276223047:84 , srv1304344444:84 , srv16081774:84 , srv1655603829:84 , srv1816622158:84 , srv203534509:84 , srv385173313:84 , srv9226263:84 , srv1125573121:85 , srv1134507230:85 , srv1182489063:85 , srv1275548031:85 , srv1412812751:85 , srv1416832389:85 , srv1556066751:85 , srv1558322610:85 , srv1603025151:85 , srv1749831202:85 , srv2000826476:85 , srv274768748:85 , srv413664870:85 , srv564159602:85 , srv588148747:85 , srv628752794:85 , srv716816614:85 , srv765499841:85 , srv791491322:85 , srv893404711:85 , srv1002956439:86 , srv1340098439:86 , srv1600449813:86 , srv1848478899:86 , srv1882434060:86 , srv1923829248:86 , srv2034890388:86 , srv207071996:86 , srv2084222789:86 , srv245037011:86 , srv717348547:86 , srv836328591:86 , srv997256211:86 , srv1060638226:87 , srv1149605709:87 , srv1480007736:87 , srv1696670394:87 , srv1998440641:87 , srv27928829:87 , srv302716415:87 , srv321342589:87 , srv358639470:87 , srv405616989:87 , srv436905221:87 , srv447746112:87 , srv603297809:87 , srv762911317:87 , srv790361937:87 , srv882847609:87 , srv937283431:87 , srv978855970:87 , srv980493794:87 , srv1019184435:88 , srv1035344488:88 , srv1055809399:88 , srv1186074813:88 , srv1829366367:88 , srv2022150065:88 , srv293222208:88 , srv403864605:88 , srv664595304:88 , srv1049019563:89 , srv1435078148:89 , srv1854626423:89 , srv1943776393:89 , srv2106613087:89 , srv222764382:89 , srv42885191:89 , srv430674973:89 , srv498874258:89 , srv747764772:89 , srv1692914487:90 , srv780052633:90 , srv1437796997:91 , srv1002245603:92 , srv306881934:92 , srv656655689:94 , srv881791326:95 , srv1591563966:452 } 2023-11-11 18:55:11,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 18:55:11,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1437796997=23, srv2084222789=50, srv762911317=84, srv245037011=54, srv1412812751=20, srv1749831202=35, srv1998440641=44, srv2034890388=47, srv451973437=70, srv1002956439=1, srv717348547=81, srv1655603829=32, srv302716415=58, srv1018746635=2, srv978855970=97, srv1854626423=39, srv747764772=82, srv1304344444=18, srv293222208=57, srv207071996=49, srv1276223047=17, srv836328591=91, srv791491322=89, srv1692914487=33, srv1829366367=37, srv1882434060=40, srv413664870=65, srv1416832389=21, srv509973775=73, srv1924243033=42, srv1943776393=43, srv16081774=30, srv1035344488=5, srv588148747=75, srv1848478899=38, srv1186074813=13, srv1019184435=3, srv1275548031=15, srv1591563966=27, srv1149605709=11, srv1055809399=7, srv1060638226=8, srv1558322610=26, srv1652092276=31, srv980493794=98, srv498874258=72, srv772055113=86, srv882847609=93, srv203534509=48, srv1049019563=6, srv1696670394=34, srv306881934=59, srv102930737=4, srv750414013=83, srv405616989=64, srv1556066751=25, srv833950709=90, srv436905221=68, srv1125573121=9, srv321342589=60, srv1600449813=28, srv430674973=67, srv1816622158=36, srv447746112=69, srv274768748=55, srv2022150065=46, srv466059494=71, srv2108926119=52, srv628752794=77, srv1002245603=0, srv42885191=66, srv1182489063=12, srv937283431=96, srv1923829248=41, srv603297809=76, srv385173313=62, srv222764382=53, srv780052633=87, srv564159602=74, srv27928829=56, srv1134507230=10, srv1275718627=16, srv765499841=85, srv2000826476=45, srv716816614=80, srv358639470=61, srv1265740510=14, srv2106613087=51, srv1340098439=19, srv1603025151=29, srv1435078148=22, srv656655689=78, srv893404711=94, srv9226263=95, srv997256211=99, srv403864605=63, srv664595304=79, srv881791326=92, srv790361937=88, srv1480007736=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 18:55:11,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 18:55:11,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 18:55:11,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 18:55:11,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 18:55:11,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 18:55:11,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 18:55:11,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 18:55:11,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 18:55:11,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 18:55:11,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 18:55:11,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 18:55:11,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 18:55:11,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 18:55:11,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 18:55:11,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 18:55:11,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 18:55:11,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 18:55:11,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 18:55:11,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 18:55:11,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 18:55:11,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 18:55:11,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 18:55:11,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 18:55:11,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 18:55:11,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 18:55:11,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 18:55:11,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 18:55:11,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 18:55:11,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 18:55:11,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 18:55:11,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-11 18:55:11,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-11 18:55:11,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-11 18:55:11,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-11 18:55:11,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-11 18:55:11,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-11 18:55:11,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-11 18:55:11,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-11 18:55:11,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-11 18:55:11,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-11 18:55:11,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-11 18:55:11,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-11 18:55:11,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-11 18:55:11,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-11 18:55:11,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-11 18:55:11,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-11 18:55:11,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-11 18:55:11,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-11 18:55:11,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-11 18:55:11,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-11 18:55:11,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-11 18:55:11,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-11 18:55:11,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-11 18:55:11,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-11 18:55:11,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-11 18:55:11,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-11 18:55:11,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-11 18:55:11,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-11 18:55:11,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-11 18:55:11,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-11 18:55:11,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-11 18:55:11,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-11 18:55:11,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-11 18:55:11,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-11 18:55:11,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-11 18:55:11,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-11 18:55:11,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-11 18:55:11,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-11 18:55:11,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-11 18:55:11,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-11 18:55:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-11 18:55:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-11 18:55:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-11 18:55:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-11 18:55:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-11 18:55:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-11 18:55:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-11 18:55:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-11 18:55:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-11 18:55:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-11 18:55:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-11 18:55:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-11 18:55:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-11 18:55:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-11 18:55:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-11 18:55:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-11 18:55:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-11 18:55:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-11 18:55:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-11 18:55:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-11 18:55:11,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-11 18:55:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-11 18:55:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-11 18:55:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-11 18:55:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-11 18:55:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-11 18:55:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-11 18:55:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-11 18:55:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-11 18:55:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-11 18:55:11,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-11 18:55:11,695 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04071777742802447, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0513520889257393, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04071777742802447, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17275, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24233333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:55:11,695 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-11 18:55:11,696 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17776671487994988, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04071777742802447, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0513520889257393, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04071777742802447, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17275, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24233333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-11 18:55:11,984 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 392 ms to try 5000 different iterations. Found a solution that moves 2648 regions; Going from a computed imbalance of 0.17776671487994988 to a new imbalance of 0.012923355768409217. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02429964253872193); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03252211541430422, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2942222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02429964253872193); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:55:11,995 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv881791326:78 , srv306881934:79 , srv656655689:80 , srv1002245603:81 , srv1437796997:81 , srv1854626423:81 , srv1035344488:82 , srv1692914487:82 , srv447746112:82 , srv498874258:82 , srv1186074813:83 , srv1998440641:83 , srv2106613087:83 , srv358639470:83 , srv403864605:83 , srv405616989:83 , srv42885191:83 , srv780052633:83 , srv1019184435:84 , srv1049019563:84 , srv1055809399:84 , srv1149605709:84 , srv1340098439:84 , srv1480007736:84 , srv1696670394:84 , srv1829366367:84 , srv762911317:84 , srv790361937:84 , srv882847609:84 , srv893404711:84 , srv978855970:84 , srv1002956439:85 , srv1416832389:85 , srv1556066751:85 , srv1600449813:85 , srv1882434060:85 , srv2034890388:85 , srv222764382:85 , srv385173313:85 , srv413664870:85 , srv430674973:85 , srv436905221:85 , srv588148747:85 , srv664595304:85 , srv716816614:85 , srv765499841:85 , srv791491322:85 , srv836328591:85 , srv1435078148:86 , srv16081774:86 , srv1943776393:86 , srv2000826476:86 , srv2022150065:86 , srv203534509:86 , srv2108926119:86 , srv293222208:86 , srv321342589:86 , srv603297809:86 , srv628752794:86 , srv747764772:86 , srv980493794:86 , srv102930737:87 , srv1060638226:87 , srv1125573121:87 , srv1265740510:87 , srv1276223047:87 , srv1412812751:87 , srv1655603829:87 , srv1749831202:87 , srv1816622158:87 , srv1848478899:87 , srv1923829248:87 , srv207071996:87 , srv245037011:87 , srv274768748:87 , srv27928829:87 , srv302716415:87 , srv509973775:87 , srv717348547:87 , srv9226263:87 , srv937283431:87 , srv1134507230:88 , srv1275548031:88 , srv1275718627:88 , srv1304344444:88 , srv1558322610:88 , srv1603025151:88 , srv1652092276:88 , srv2084222789:88 , srv451973437:88 , srv564159602:88 , srv833950709:88 , srv997256211:88 , srv1182489063:89 , srv750414013:89 , srv772055113:89 , srv1018746635:90 , srv1924243033:91 , srv466059494:91 , srv1591563966:538 } 2023-11-11 18:55:11,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 18:55:11,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1437796997=23, srv2084222789=50, srv762911317=84, srv245037011=54, srv1412812751=20, srv1749831202=35, srv1998440641=44, srv2034890388=47, srv451973437=70, srv1002956439=1, srv717348547=81, srv1655603829=32, srv302716415=58, srv1018746635=2, srv978855970=97, srv1854626423=39, srv747764772=82, srv1304344444=18, srv293222208=57, srv207071996=49, srv1276223047=17, srv836328591=91, srv791491322=89, srv1692914487=33, srv1829366367=37, srv1882434060=40, srv413664870=65, srv1416832389=21, srv509973775=73, srv1924243033=42, srv1943776393=43, srv16081774=30, srv1035344488=5, srv588148747=75, srv1848478899=38, srv1186074813=13, srv1019184435=3, srv1275548031=15, srv1591563966=27, srv1149605709=11, srv1055809399=7, srv1060638226=8, srv1558322610=26, srv1652092276=31, srv980493794=98, srv498874258=72, srv772055113=86, srv882847609=93, srv203534509=48, srv1049019563=6, srv1696670394=34, srv306881934=59, srv102930737=4, srv750414013=83, srv405616989=64, srv1556066751=25, srv833950709=90, srv436905221=68, srv1125573121=9, srv321342589=60, srv1600449813=28, srv430674973=67, srv1816622158=36, srv447746112=69, srv274768748=55, srv2022150065=46, srv466059494=71, srv2108926119=52, srv628752794=77, srv1002245603=0, srv42885191=66, srv1182489063=12, srv937283431=96, srv1923829248=41, srv603297809=76, srv385173313=62, srv222764382=53, srv780052633=87, srv564159602=74, srv27928829=56, srv1134507230=10, srv1275718627=16, srv765499841=85, srv2000826476=45, srv716816614=80, srv358639470=61, srv1265740510=14, srv2106613087=51, srv1340098439=19, srv1603025151=29, srv1435078148=22, srv656655689=78, srv893404711=94, srv9226263=95, srv997256211=99, srv403864605=63, srv664595304=79, srv881791326=92, srv790361937=88, srv1480007736=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 18:55:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 18:55:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 18:55:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 18:55:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 18:55:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 18:55:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 18:55:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 18:55:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 18:55:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 18:55:12,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 18:55:12,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-11 18:55:12,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-11 18:55:12,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-11 18:55:12,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-11 18:55:12,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-11 18:55:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-11 18:55:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-11 18:55:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-11 18:55:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-11 18:55:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-11 18:55:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-11 18:55:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-11 18:55:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-11 18:55:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-11 18:55:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-11 18:55:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-11 18:55:12,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-11 18:55:12,130 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02429964253872193); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03252211541430422, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02429964253872193); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:55:12,131 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-11 18:55:12,131 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.012904814076109682, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02429964253872193); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03252211541430422, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02429964253872193); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.009583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-11 18:55:12,444 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 413 ms to try 5000 different iterations. Found a solution that moves 1008 regions; Going from a computed imbalance of 0.012904814076109682 to a new imbalance of 0.001075881136214296. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071137060360810895); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008164965809277261); MoveCostFunction : (multiplier=7.0, imbalance=0.112, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0071137060360810895); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:55:12,446 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1002245603:83 , srv1060638226:83 , srv1435078148:83 , srv293222208:83 , srv430674973:83 , srv466059494:83 , srv747764772:83 , srv937283431:83 , srv997256211:83 , srv1018746635:84 , srv1049019563:84 , srv1134507230:84 , srv1182489063:84 , srv1275548031:84 , srv1412812751:84 , srv1558322610:84 , srv1943776393:84 , srv2022150065:84 , srv207071996:84 , srv2084222789:84 , srv222764382:84 , srv27928829:84 , srv321342589:84 , srv603297809:84 , srv656655689:84 , srv664595304:84 , srv881791326:84 , srv1019184435:85 , srv1125573121:85 , srv1149605709:85 , srv1304344444:85 , srv1603025151:85 , srv1655603829:85 , srv1692914487:85 , srv1749831202:85 , srv1829366367:85 , srv1848478899:85 , srv1923829248:85 , srv2034890388:85 , srv2106613087:85 , srv245037011:85 , srv274768748:85 , srv302716415:85 , srv436905221:85 , srv564159602:85 , srv717348547:85 , srv750414013:85 , srv980493794:85 , srv1002956439:86 , srv102930737:86 , srv1055809399:86 , srv1186074813:86 , srv1265740510:86 , srv1275718627:86 , srv1437796997:86 , srv1600449813:86 , srv1652092276:86 , srv1816622158:86 , srv1924243033:86 , srv306881934:86 , srv42885191:86 , srv451973437:86 , srv498874258:86 , srv588148747:86 , srv628752794:86 , srv762911317:86 , srv765499841:86 , srv772055113:86 , srv780052633:86 , srv790361937:86 , srv9226263:86 , srv1035344488:87 , srv1276223047:87 , srv1340098439:87 , srv1480007736:87 , srv16081774:87 , srv1696670394:87 , srv1854626423:87 , srv2000826476:87 , srv203534509:87 , srv2108926119:87 , srv403864605:87 , srv413664870:87 , srv509973775:87 , srv716816614:87 , srv791491322:87 , srv833950709:87 , srv836328591:87 , srv882847609:87 , srv978855970:87 , srv1416832389:88 , srv1556066751:88 , srv1882434060:88 , srv1998440641:88 , srv385173313:88 , srv405616989:88 , srv447746112:88 , srv358639470:89 , srv893404711:89 , srv1591563966:531 } 2023-11-11 18:55:12,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 18:55:12,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1437796997=23, srv2084222789=50, srv762911317=84, srv245037011=54, srv1412812751=20, srv1749831202=35, srv1998440641=44, srv2034890388=47, srv451973437=70, srv1002956439=1, srv717348547=81, srv1655603829=32, srv302716415=58, srv1018746635=2, srv978855970=97, srv1854626423=39, srv747764772=82, srv1304344444=18, srv293222208=57, srv207071996=49, srv1276223047=17, srv836328591=91, srv791491322=89, srv1692914487=33, srv1829366367=37, srv1882434060=40, srv413664870=65, srv1416832389=21, srv509973775=73, srv1924243033=42, srv1943776393=43, srv16081774=30, srv1035344488=5, srv588148747=75, srv1848478899=38, srv1186074813=13, srv1019184435=3, srv1275548031=15, srv1591563966=27, srv1149605709=11, srv1055809399=7, srv1060638226=8, srv1558322610=26, srv1652092276=31, srv980493794=98, srv498874258=72, srv772055113=86, srv882847609=93, srv203534509=48, srv1049019563=6, srv1696670394=34, srv306881934=59, srv102930737=4, srv750414013=83, srv405616989=64, srv1556066751=25, srv833950709=90, srv436905221=68, srv1125573121=9, srv321342589=60, srv1600449813=28, srv430674973=67, srv1816622158=36, srv447746112=69, srv274768748=55, srv2022150065=46, srv466059494=71, srv2108926119=52, srv628752794=77, srv1002245603=0, srv42885191=66, srv1182489063=12, srv937283431=96, srv1923829248=41, srv603297809=76, srv385173313=62, srv222764382=53, srv780052633=87, srv564159602=74, srv27928829=56, srv1134507230=10, srv1275718627=16, srv765499841=85, srv2000826476=45, srv716816614=80, srv358639470=61, srv1265740510=14, srv2106613087=51, srv1340098439=19, srv1603025151=29, srv1435078148=22, srv656655689=78, srv893404711=94, srv9226263=95, srv997256211=99, srv403864605=63, srv664595304=79, srv881791326=92, srv790361937=88, srv1480007736=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 18:55:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 18:55:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 18:55:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 18:55:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 18:55:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 18:55:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 18:55:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 18:55:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 18:55:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 18:55:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 18:55:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 18:55:12,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 18:55:12,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-11 18:55:12,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-11 18:55:12,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-11 18:55:12,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-11 18:55:12,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-11 18:55:12,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-11 18:55:12,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-11 18:55:12,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-11 18:55:12,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-11 18:55:12,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-11 18:55:12,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-11 18:55:12,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-11 18:55:12,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-11 18:55:12,576 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071137060360810895); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008164965809277261); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0071137060360810895); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:55:12,576 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-11 18:55:12,576 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0010688229693570706, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0071137060360810895); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008164965809277261); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0071137060360810895); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00525); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-11 18:55:12,935 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 456 ms to try 5000 different iterations. Found a solution that moves 277 regions; Going from a computed imbalance of 0.0010688229693570706 to a new imbalance of 4.860056517425982E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.03077777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:55:12,936 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1943776393:82 , srv1055809399:83 , srv1882434060:83 , srv302716415:83 , srv358639470:83 , srv780052633:83 , srv1182489063:84 , srv1275548031:84 , srv1276223047:84 , srv1340098439:84 , srv1416832389:84 , srv1437796997:84 , srv1480007736:84 , srv1556066751:84 , srv1600449813:84 , srv1603025151:84 , srv1652092276:84 , srv1696670394:84 , srv1848478899:84 , srv1854626423:84 , srv1923829248:84 , srv2000826476:84 , srv2022150065:84 , srv2084222789:84 , srv222764382:84 , srv27928829:84 , srv403864605:84 , srv405616989:84 , srv42885191:84 , srv564159602:84 , srv656655689:84 , srv664595304:84 , srv717348547:84 , srv747764772:84 , srv772055113:84 , srv836328591:84 , srv882847609:84 , srv893404711:84 , srv978855970:84 , srv980493794:84 , srv1002956439:85 , srv1018746635:85 , srv1019184435:85 , srv1035344488:85 , srv1060638226:85 , srv1125573121:85 , srv1134507230:85 , srv1186074813:85 , srv1265740510:85 , srv1275718627:85 , srv1435078148:85 , srv1558322610:85 , srv16081774:85 , srv1692914487:85 , srv1749831202:85 , srv1816622158:85 , srv1829366367:85 , srv1924243033:85 , srv1998440641:85 , srv2034890388:85 , srv203534509:85 , srv207071996:85 , srv2106613087:85 , srv2108926119:85 , srv245037011:85 , srv274768748:85 , srv293222208:85 , srv306881934:85 , srv385173313:85 , srv413664870:85 , srv430674973:85 , srv436905221:85 , srv447746112:85 , srv451973437:85 , srv466059494:85 , srv498874258:85 , srv509973775:85 , srv603297809:85 , srv628752794:85 , srv716816614:85 , srv750414013:85 , srv762911317:85 , srv790361937:85 , srv791491322:85 , srv833950709:85 , srv9226263:85 , srv937283431:85 , srv997256211:85 , srv1002245603:86 , srv102930737:86 , srv1049019563:86 , srv1149605709:86 , srv1304344444:86 , srv1412812751:86 , srv1655603829:86 , srv321342589:86 , srv588148747:86 , srv765499841:86 , srv881791326:86 , srv1591563966:621 } 2023-11-11 18:55:12,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 18:55:12,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1437796997=23, srv2084222789=50, srv762911317=84, srv245037011=54, srv1412812751=20, srv1749831202=35, srv1998440641=44, srv2034890388=47, srv451973437=70, srv1002956439=1, srv717348547=81, srv1655603829=32, srv302716415=58, srv1018746635=2, srv978855970=97, srv1854626423=39, srv747764772=82, srv1304344444=18, srv293222208=57, srv207071996=49, srv1276223047=17, srv836328591=91, srv791491322=89, srv1692914487=33, srv1829366367=37, srv1882434060=40, srv413664870=65, srv1416832389=21, srv509973775=73, srv1924243033=42, srv1943776393=43, srv16081774=30, srv1035344488=5, srv588148747=75, srv1848478899=38, srv1186074813=13, srv1019184435=3, srv1275548031=15, srv1591563966=27, srv1149605709=11, srv1055809399=7, srv1060638226=8, srv1558322610=26, srv1652092276=31, srv980493794=98, srv498874258=72, srv772055113=86, srv882847609=93, srv203534509=48, srv1049019563=6, srv1696670394=34, srv306881934=59, srv102930737=4, srv750414013=83, srv405616989=64, srv1556066751=25, srv833950709=90, srv436905221=68, srv1125573121=9, srv321342589=60, srv1600449813=28, srv430674973=67, srv1816622158=36, srv447746112=69, srv274768748=55, srv2022150065=46, srv466059494=71, srv2108926119=52, srv628752794=77, srv1002245603=0, srv42885191=66, srv1182489063=12, srv937283431=96, srv1923829248=41, srv603297809=76, srv385173313=62, srv222764382=53, srv780052633=87, srv564159602=74, srv27928829=56, srv1134507230=10, srv1275718627=16, srv765499841=85, srv2000826476=45, srv716816614=80, srv358639470=61, srv1265740510=14, srv2106613087=51, srv1340098439=19, srv1603025151=29, srv1435078148=22, srv656655689=78, srv893404711=94, srv9226263=95, srv997256211=99, srv403864605=63, srv664595304=79, srv881791326=92, srv790361937=88, srv1480007736=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 18:55:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 18:55:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 18:55:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 18:55:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 18:55:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 18:55:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 18:55:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-11 18:55:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-11 18:55:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-11 18:55:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-11 18:55:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-11 18:55:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-11 18:55:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-11 18:55:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-11 18:55:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-11 18:55:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-11 18:55:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-11 18:55:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-11 18:55:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-11 18:55:13,065 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:55:13,065 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-11 18:55:13,066 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.6660969718454885E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-11 18:55:13,327 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 362 ms to try 5000 different iterations. Found a solution that moves 28 regions; Going from a computed imbalance of 4.6660969718454885E-5 to a new imbalance of 5.419993616711265E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0011605177063713189); MoveCostFunction : (multiplier=7.0, imbalance=0.003111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:55:13,328 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv102930737:83 , srv1049019563:83 , srv1304344444:83 , srv1435078148:83 , srv1924243033:83 , srv1998440641:83 , srv245037011:83 , srv321342589:83 , srv833950709:83 , srv881791326:83 , srv1002245603:84 , srv1002956439:84 , srv1018746635:84 , srv1019184435:84 , srv1035344488:84 , srv1060638226:84 , srv1125573121:84 , srv1134507230:84 , srv1149605709:84 , srv1182489063:84 , srv1186074813:84 , srv1265740510:84 , srv1275718627:84 , srv1276223047:84 , srv1412812751:84 , srv1416832389:84 , srv1437796997:84 , srv1480007736:84 , srv1556066751:84 , srv1558322610:84 , srv1603025151:84 , srv16081774:84 , srv1655603829:84 , srv1692914487:84 , srv1696670394:84 , srv1749831202:84 , srv1816622158:84 , srv1829366367:84 , srv1848478899:84 , srv1882434060:84 , srv1923829248:84 , srv2000826476:84 , srv2022150065:84 , srv203534509:84 , srv207071996:84 , srv2084222789:84 , srv2106613087:84 , srv222764382:84 , srv274768748:84 , srv27928829:84 , srv293222208:84 , srv302716415:84 , srv306881934:84 , srv358639470:84 , srv385173313:84 , srv403864605:84 , srv405616989:84 , srv413664870:84 , srv42885191:84 , srv430674973:84 , srv436905221:84 , srv447746112:84 , srv451973437:84 , srv466059494:84 , srv498874258:84 , srv509973775:84 , srv564159602:84 , srv588148747:84 , srv603297809:84 , srv628752794:84 , srv656655689:84 , srv716816614:84 , srv717348547:84 , srv747764772:84 , srv750414013:84 , srv762911317:84 , srv765499841:84 , srv772055113:84 , srv780052633:84 , srv790361937:84 , srv791491322:84 , srv836328591:84 , srv882847609:84 , srv893404711:84 , srv9226263:84 , srv937283431:84 , srv978855970:84 , srv980493794:84 , srv997256211:84 , srv1055809399:85 , srv1275548031:85 , srv1340098439:85 , srv1600449813:85 , srv1652092276:85 , srv1854626423:85 , srv1943776393:85 , srv2034890388:85 , srv2108926119:85 , srv664595304:85 , srv1591563966:684 } 2023-11-11 18:55:13,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 18:55:13,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1437796997=23, srv2084222789=50, srv762911317=84, srv245037011=54, srv1412812751=20, srv1749831202=35, srv1998440641=44, srv2034890388=47, srv451973437=70, srv1002956439=1, srv717348547=81, srv1655603829=32, srv302716415=58, srv1018746635=2, srv978855970=97, srv1854626423=39, srv747764772=82, srv1304344444=18, srv293222208=57, srv207071996=49, srv1276223047=17, srv836328591=91, srv791491322=89, srv1692914487=33, srv1829366367=37, srv1882434060=40, srv413664870=65, srv1416832389=21, srv509973775=73, srv1924243033=42, srv1943776393=43, srv16081774=30, srv1035344488=5, srv588148747=75, srv1848478899=38, srv1186074813=13, srv1019184435=3, srv1275548031=15, srv1591563966=27, srv1149605709=11, srv1055809399=7, srv1060638226=8, srv1558322610=26, srv1652092276=31, srv980493794=98, srv498874258=72, srv772055113=86, srv882847609=93, srv203534509=48, srv1049019563=6, srv1696670394=34, srv306881934=59, srv102930737=4, srv750414013=83, srv405616989=64, srv1556066751=25, srv833950709=90, srv436905221=68, srv1125573121=9, srv321342589=60, srv1600449813=28, srv430674973=67, srv1816622158=36, srv447746112=69, srv274768748=55, srv2022150065=46, srv466059494=71, srv2108926119=52, srv628752794=77, srv1002245603=0, srv42885191=66, srv1182489063=12, srv937283431=96, srv1923829248=41, srv603297809=76, srv385173313=62, srv222764382=53, srv780052633=87, srv564159602=74, srv27928829=56, srv1134507230=10, srv1275718627=16, srv765499841=85, srv2000826476=45, srv716816614=80, srv358639470=61, srv1265740510=14, srv2106613087=51, srv1340098439=19, srv1603025151=29, srv1435078148=22, srv656655689=78, srv893404711=94, srv9226263=95, srv997256211=99, srv403864605=63, srv664595304=79, srv881791326=92, srv790361937=88, srv1480007736=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 18:55:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 18:55:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 18:55:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 18:55:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 18:55:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 18:55:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 18:55:13,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 18:55:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 18:55:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 18:55:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 18:55:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 18:55:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 18:55:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 18:55:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 18:55:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 18:55:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 18:55:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 18:55:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 18:55:13,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 18:55:13,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 18:55:13,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 18:55:13,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 18:55:13,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 18:55:13,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 18:55:13,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 18:55:13,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 18:55:13,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 18:55:13,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 18:55:13,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 18:55:13,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 18:55:13,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-11 18:55:13,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-11 18:55:13,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-11 18:55:13,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-11 18:55:13,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-11 18:55:13,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-11 18:55:13,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-11 18:55:13,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-11 18:55:13,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-11 18:55:13,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-11 18:55:13,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-11 18:55:13,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-11 18:55:13,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-11 18:55:13,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-11 18:55:13,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-11 18:55:13,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-11 18:55:13,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-11 18:55:13,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-11 18:55:13,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-11 18:55:13,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-11 18:55:13,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-11 18:55:13,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-11 18:55:13,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-11 18:55:13,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-11 18:55:13,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-11 18:55:13,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-11 18:55:13,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-11 18:55:13,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-11 18:55:13,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-11 18:55:13,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-11 18:55:13,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-11 18:55:13,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-11 18:55:13,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-11 18:55:13,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-11 18:55:13,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-11 18:55:13,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-11 18:55:13,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-11 18:55:13,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-11 18:55:13,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-11 18:55:13,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-11 18:55:13,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-11 18:55:13,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-11 18:55:13,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-11 18:55:13,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-11 18:55:13,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-11 18:55:13,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-11 18:55:13,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-11 18:55:13,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-11 18:55:13,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-11 18:55:13,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-11 18:55:13,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-11 18:55:13,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-11 18:55:13,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-11 18:55:13,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-11 18:55:13,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-11 18:55:13,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-11 18:55:13,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-11 18:55:13,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-11 18:55:13,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-11 18:55:13,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-11 18:55:13,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-11 18:55:13,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-11 18:55:13,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-11 18:55:13,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-11 18:55:13,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-11 18:55:13,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-11 18:55:13,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-11 18:55:13,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-11 18:55:13,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-11 18:55:13,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-11 18:55:13,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-11 18:55:13,467 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=5.223933426232788E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0011605177063713189); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:55:13,467 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv102930737:83 , srv1049019563:83 , srv1304344444:83 , srv1435078148:83 , srv1924243033:83 , srv1998440641:83 , srv245037011:83 , srv321342589:83 , srv833950709:83 , srv881791326:83 , srv1002245603:84 , srv1002956439:84 , srv1018746635:84 , srv1019184435:84 , srv1035344488:84 , srv1060638226:84 , srv1125573121:84 , srv1134507230:84 , srv1149605709:84 , srv1182489063:84 , srv1186074813:84 , srv1265740510:84 , srv1275718627:84 , srv1276223047:84 , srv1412812751:84 , srv1416832389:84 , srv1437796997:84 , srv1480007736:84 , srv1556066751:84 , srv1558322610:84 , srv1603025151:84 , srv16081774:84 , srv1655603829:84 , srv1692914487:84 , srv1696670394:84 , srv1749831202:84 , srv1816622158:84 , srv1829366367:84 , srv1848478899:84 , srv1882434060:84 , srv1923829248:84 , srv2000826476:84 , srv2022150065:84 , srv203534509:84 , srv207071996:84 , srv2084222789:84 , srv2106613087:84 , srv222764382:84 , srv274768748:84 , srv27928829:84 , srv293222208:84 , srv302716415:84 , srv306881934:84 , srv358639470:84 , srv385173313:84 , srv403864605:84 , srv405616989:84 , srv413664870:84 , srv42885191:84 , srv430674973:84 , srv436905221:84 , srv447746112:84 , srv451973437:84 , srv466059494:84 , srv498874258:84 , srv509973775:84 , srv564159602:84 , srv588148747:84 , srv603297809:84 , srv628752794:84 , srv656655689:84 , srv716816614:84 , srv717348547:84 , srv747764772:84 , srv750414013:84 , srv762911317:84 , srv765499841:84 , srv772055113:84 , srv780052633:84 , srv790361937:84 , srv791491322:84 , srv836328591:84 , srv882847609:84 , srv893404711:84 , srv9226263:84 , srv937283431:84 , srv978855970:84 , srv980493794:84 , srv997256211:84 , srv1055809399:85 , srv1275548031:85 , srv1340098439:85 , srv1600449813:85 , srv1652092276:85 , srv1854626423:85 , srv1943776393:85 , srv2034890388:85 , srv2108926119:85 , srv664595304:85 , srv1591563966:684 } 2023-11-11 18:55:13,491 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=107 (was 107), ProcessCount=167 (was 167), AvailableMemoryMB=1112 (was 1615) 2023-11-11 18:55:13,499 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=107, ProcessCount=167, AvailableMemoryMB=1112 2023-11-11 18:55:13,500 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-11 18:55:13,500 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-11-11 18:55:13,512 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv107197683:84 , srv1128171337:84 , srv1168979153:84 , srv1206589813:84 , srv1244743678:84 , srv1298733896:84 , srv1360013171:84 , srv1469282870:84 , srv1567629567:84 , srv1601969751:84 , srv1612982985:84 , srv1755156528:84 , srv1766984431:84 , srv1807046848:84 , srv1892108518:84 , srv1981959717:84 , srv2078262908:84 , srv233335407:84 , srv274424270:84 , srv325453374:84 , srv379323148:84 , srv441758340:84 , srv518297314:84 , srv568568332:84 , srv715871372:84 , srv735007026:84 , srv738941725:84 , srv740788997:84 , srv989967898:84 , srv1136229376:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-11-11 18:55:13,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 18:55:13,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv735007026=26, srv107197683=0, srv1612982985=11, srv233335407=18, srv2078262908=17, srv1136229376=2, srv568568332=24, srv1128171337=1, srv989967898=29, srv325453374=20, srv1807046848=14, srv1298733896=6, srv1469282870=8, srv274424270=19, srv379323148=21, srv1168979153=3, srv518297314=23, srv715871372=25, srv1892108518=15, srv1244743678=5, srv1567629567=9, srv1755156528=12, srv738941725=27, srv1981959717=16, srv1206589813=4, srv1601969751=10, srv740788997=28, srv1360013171=7, srv1766984431=13, srv441758340=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 18:55:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 18:55:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 18:55:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 18:55:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 18:55:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 18:55:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 18:55:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 18:55:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 18:55:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 18:55:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 18:55:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 18:55:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 18:55:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 18:55:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 18:55:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 18:55:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 18:55:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 18:55:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 18:55:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 18:55:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 18:55:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 18:55:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 18:55:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 18:55:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 18:55:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 18:55:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 18:55:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 18:55:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 18:55:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 18:55:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 18:55:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-11-11 18:55:13,556 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.169172667605022, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:55:13,556 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912726100815906, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.169172667605022, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-11-11 18:57:13,535 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5487764 different iterations. Found a solution that moves 1862 regions; Going from a computed imbalance of 0.9912726100815906 to a new imbalance of 4.346000888939571E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6896296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:57:13,542 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv107197683:90 , srv1128171337:90 , srv1136229376:90 , srv1168979153:90 , srv1206589813:90 , srv1244743678:90 , srv1298733896:90 , srv1360013171:90 , srv1469282870:90 , srv1567629567:90 , srv1601969751:90 , srv1612982985:90 , srv1755156528:90 , srv1766984431:90 , srv1807046848:90 , srv1892108518:90 , srv1981959717:90 , srv2078262908:90 , srv233335407:90 , srv274424270:90 , srv325453374:90 , srv379323148:90 , srv441758340:90 , srv518297314:90 , srv568568332:90 , srv715871372:90 , srv735007026:90 , srv738941725:90 , srv740788997:90 , srv989967898:90 } 2023-11-11 18:57:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 18:57:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv735007026=26, srv107197683=0, srv1612982985=11, srv233335407=18, srv2078262908=17, srv1136229376=2, srv568568332=24, srv1128171337=1, srv989967898=29, srv325453374=20, srv1807046848=14, srv1298733896=6, srv1469282870=8, srv274424270=19, srv379323148=21, srv1168979153=3, srv518297314=23, srv715871372=25, srv1892108518=15, srv1244743678=5, srv1567629567=9, srv1755156528=12, srv738941725=27, srv1981959717=16, srv1206589813=4, srv1601969751=10, srv740788997=28, srv1360013171=7, srv1766984431=13, srv441758340=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 18:57:13,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 18:57:13,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 18:57:13,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 18:57:13,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 18:57:13,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 18:57:13,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 18:57:13,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 18:57:13,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 18:57:13,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-11-11 18:57:13,560 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 18:57:13,560 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv107197683:90 , srv1128171337:90 , srv1136229376:90 , srv1168979153:90 , srv1206589813:90 , srv1244743678:90 , srv1298733896:90 , srv1360013171:90 , srv1469282870:90 , srv1567629567:90 , srv1601969751:90 , srv1612982985:90 , srv1755156528:90 , srv1766984431:90 , srv1807046848:90 , srv1892108518:90 , srv1981959717:90 , srv2078262908:90 , srv233335407:90 , srv274424270:90 , srv325453374:90 , srv379323148:90 , srv441758340:90 , srv518297314:90 , srv568568332:90 , srv715871372:90 , srv735007026:90 , srv738941725:90 , srv740788997:90 , srv989967898:90 } 2023-11-11 18:57:13,572 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=288 (was 107) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=1302 (was 1112) - AvailableMemoryMB LEAK? -