2024-02-01 05:53:11,831 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-02-01 05:53:12,136 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-02-01 05:53:12,251 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-02-01 05:53:12,324 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-01 05:53:12,333 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-01 05:53:12,350 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=333, ProcessCount=167, AvailableMemoryMB=3151 2024-02-01 05:53:12,352 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-01 05:53:12,353 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-01 05:53:12,480 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1018510925:84 , srv1026963014:84 , srv1049335747:84 , srv1061972837:84 , srv1080071534:84 , srv1104237660:84 , srv1109218513:84 , srv1156746682:84 , srv1200921403:84 , srv122406504:84 , srv1233361248:84 , srv1253192618:84 , srv1257968344:84 , srv1275523832:84 , srv130096813:84 , srv1309142990:84 , srv1321796152:84 , srv1329761749:84 , srv137739713:84 , srv1382502663:84 , srv1386855364:84 , srv1408770410:84 , srv1422069607:84 , srv1436216877:84 , srv1467513298:84 , srv1468006260:84 , srv1472282925:84 , srv1477582655:84 , srv1478126384:84 , srv1483265914:84 , srv1555052259:84 , srv1562800769:84 , srv1570910827:84 , srv1592016503:84 , srv1665609579:84 , srv1671713106:84 , srv1726792032:84 , srv1732205895:84 , srv1744236186:84 , srv1752418177:84 , srv1760688380:84 , srv1762938620:84 , srv1764140497:84 , srv1776623870:84 , srv1786134674:84 , srv1792577933:84 , srv180321322:84 , srv1814763691:84 , srv1826652184:84 , srv1841293332:84 , srv1863862330:84 , srv1878613400:84 , srv1885864467:84 , srv1888637889:84 , srv1930816390:84 , srv1958797122:84 , srv1975989725:84 , srv1991516990:84 , srv1994955403:84 , srv2040580910:84 , srv2118469471:84 , srv2127502907:84 , srv223074031:84 , srv278425528:84 , srv286482742:84 , srv288693666:84 , srv294577225:84 , srv305857995:84 , srv342051211:84 , srv375431640:84 , srv43977079:84 , srv472097156:84 , srv473869728:84 , srv476812482:84 , srv526204737:84 , srv526496352:84 , srv589573107:84 , srv593474482:84 , srv633057988:84 , srv652705982:84 , srv684963208:84 , srv71737410:84 , srv727363266:84 , srv739393629:84 , srv750359641:84 , srv782755101:84 , srv784388946:84 , srv791712926:84 , srv795682722:84 , srv838736206:84 , srv852894486:84 , srv854413237:84 , srv91646931:84 , srv923154683:84 , srv945944631:84 , srv956716333:84 , srv966526417:84 , srv974599524:84 , srv994459734:84 , srv988834822:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-02-01 05:53:12,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 05:53:12,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1930816390=54, srv2118469471=60, srv1018510925=0, srv1975989725=56, srv795682722=88, srv1049335747=2, srv684963208=80, srv838736206=89, srv375431640=69, srv1478126384=28, srv1109218513=6, srv1200921403=8, srv288693666=65, srv1061972837=3, srv852894486=90, srv1762938620=41, srv1888637889=53, srv956716333=95, srv1253192618=11, srv1472282925=26, srv784388946=86, srv1885864467=52, srv180321322=46, srv1991516990=57, srv1104237660=5, srv43977079=70, srv286482742=64, srv1467513298=24, srv750359641=84, srv1826652184=48, srv223074031=62, srv476812482=73, srv1562800769=31, srv1386855364=20, srv1744236186=38, srv1026963014=1, srv589573107=76, srv727363266=82, srv1592016503=33, srv782755101=85, srv130096813=14, srv342051211=68, srv91646931=92, srv1726792032=36, srv1841293332=49, srv1814763691=47, srv1275523832=13, srv966526417=96, srv1665609579=34, srv1408770410=21, srv1671713106=35, srv945944631=94, srv526204737=74, srv1477582655=27, srv278425528=63, srv1752418177=39, srv1382502663=19, srv1764140497=42, srv994459734=99, srv473869728=72, srv1233361248=10, srv294577225=66, srv1776623870=43, srv1786134674=44, srv1760688380=40, srv1555052259=30, srv526496352=75, srv988834822=98, srv593474482=77, srv137739713=18, srv1732205895=37, srv1436216877=23, srv1422069607=22, srv739393629=83, srv1309142990=15, srv1156746682=7, srv1792577933=45, srv1468006260=25, srv305857995=67, srv1878613400=51, srv974599524=97, srv1329761749=17, srv1863862330=50, srv2127502907=61, srv791712926=87, srv71737410=81, srv122406504=9, srv472097156=71, srv854413237=91, srv1080071534=4, srv1994955403=58, srv1321796152=16, srv1257968344=12, srv1570910827=32, srv2040580910=59, srv652705982=79, srv923154683=93, srv633057988=78, srv1483265914=29, srv1958797122=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 05:53:12,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 05:53:12,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 05:53:12,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 05:53:12,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 05:53:12,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 05:53:12,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 05:53:12,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 05:53:12,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 05:53:12,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 05:53:12,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 05:53:12,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 05:53:12,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 05:53:12,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 05:53:12,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 05:53:12,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 05:53:12,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 05:53:12,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 05:53:12,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 05:53:12,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 05:53:12,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 05:53:12,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 05:53:12,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 05:53:12,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 05:53:12,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 05:53:12,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 05:53:12,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 05:53:12,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 05:53:12,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 05:53:12,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 05:53:12,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 05:53:12,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-01 05:53:12,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-01 05:53:12,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-01 05:53:12,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-01 05:53:12,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-01 05:53:12,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-01 05:53:12,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-01 05:53:12,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-01 05:53:12,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-01 05:53:12,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-01 05:53:12,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-01 05:53:12,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-01 05:53:12,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-01 05:53:12,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-01 05:53:12,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-01 05:53:12,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-01 05:53:12,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-01 05:53:12,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-01 05:53:12,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-01 05:53:12,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-01 05:53:12,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-01 05:53:12,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-01 05:53:12,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-01 05:53:12,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-01 05:53:12,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-01 05:53:12,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-01 05:53:12,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-01 05:53:12,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-01 05:53:12,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-01 05:53:12,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-01 05:53:12,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-01 05:53:12,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-01 05:53:12,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-01 05:53:12,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-01 05:53:12,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-01 05:53:12,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-01 05:53:12,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-01 05:53:12,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-01 05:53:12,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-01 05:53:12,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-01 05:53:12,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-01 05:53:12,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-01 05:53:12,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-01 05:53:12,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-01 05:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-01 05:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-01 05:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-01 05:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-01 05:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-01 05:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-01 05:53:12,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-01 05:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-01 05:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-01 05:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-01 05:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-01 05:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-01 05:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-01 05:53:12,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-01 05:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-01 05:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-01 05:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-01 05:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-01 05:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-01 05:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-01 05:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-01 05:53:12,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-01 05:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-01 05:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-01 05:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-01 05:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-01 05:53:12,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-01 05:53:12,854 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:53:12,854 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-01 05:53:12,855 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-01 05:53:13,232 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 576 ms to try 5000 different iterations. Found a solution that moves 4702 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17620762190148762. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03949992936074289, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05731555593152367, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5224444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03949992936074289, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17191666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:53:13,249 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1863862330:82 , srv1080071534:83 , srv1467513298:83 , srv1888637889:83 , srv223074031:83 , srv375431640:83 , srv1026963014:84 , srv1200921403:84 , srv1257968344:84 , srv1309142990:84 , srv1329761749:84 , srv1408770410:84 , srv1468006260:84 , srv1472282925:84 , srv1477582655:84 , srv1726792032:84 , srv1732205895:84 , srv1762938620:84 , srv1764140497:84 , srv1786134674:84 , srv1814763691:84 , srv1930816390:84 , srv1975989725:84 , srv472097156:84 , srv791712926:84 , srv1061972837:85 , srv1321796152:85 , srv1483265914:85 , srv1570910827:85 , srv1776623870:85 , srv1841293332:85 , srv1958797122:85 , srv1991516990:85 , srv2118469471:85 , srv286482742:85 , srv294577225:85 , srv476812482:85 , srv526204737:85 , srv633057988:85 , srv727363266:85 , srv782755101:85 , srv838736206:85 , srv945944631:85 , srv1018510925:86 , srv1109218513:86 , srv122406504:86 , srv137739713:86 , srv1422069607:86 , srv1744236186:86 , srv1752418177:86 , srv652705982:86 , srv739393629:86 , srv91646931:86 , srv923154683:86 , srv974599524:86 , srv994459734:86 , srv1104237660:87 , srv1156746682:87 , srv1233361248:87 , srv1253192618:87 , srv1275523832:87 , srv1382502663:87 , srv1478126384:87 , srv1555052259:87 , srv1885864467:87 , srv1994955403:87 , srv2040580910:87 , srv305857995:87 , srv473869728:87 , srv589573107:87 , srv593474482:87 , srv784388946:87 , srv795682722:87 , srv1562800769:88 , srv1592016503:88 , srv1760688380:88 , srv1792577933:88 , srv1826652184:88 , srv526496352:88 , srv684963208:88 , srv852894486:88 , srv956716333:88 , srv1049335747:89 , srv1671713106:89 , srv278425528:89 , srv71737410:89 , srv180321322:90 , srv1878613400:90 , srv43977079:90 , srv750359641:90 , srv130096813:91 , srv1665609579:91 , srv854413237:91 , srv966526417:91 , srv1436216877:92 , srv288693666:92 , srv342051211:93 , srv1386855364:95 , srv2127502907:95 , srv988834822:441 } 2024-02-01 05:53:13,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 05:53:13,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1930816390=54, srv2118469471=60, srv1018510925=0, srv1975989725=56, srv795682722=88, srv1049335747=2, srv684963208=80, srv838736206=89, srv375431640=69, srv1478126384=28, srv1109218513=6, srv1200921403=8, srv288693666=65, srv1061972837=3, srv852894486=90, srv1762938620=41, srv1888637889=53, srv956716333=95, srv1253192618=11, srv1472282925=26, srv784388946=86, srv1885864467=52, srv180321322=46, srv1991516990=57, srv1104237660=5, srv43977079=70, srv286482742=64, srv1467513298=24, srv750359641=84, srv1826652184=48, srv223074031=62, srv476812482=73, srv1562800769=31, srv1386855364=20, srv1744236186=38, srv1026963014=1, srv589573107=76, srv727363266=82, srv1592016503=33, srv782755101=85, srv130096813=14, srv342051211=68, srv91646931=92, srv1726792032=36, srv1841293332=49, srv1814763691=47, srv1275523832=13, srv966526417=96, srv1665609579=34, srv1408770410=21, srv1671713106=35, srv945944631=94, srv526204737=74, srv1477582655=27, srv278425528=63, srv1752418177=39, srv1382502663=19, srv1764140497=42, srv994459734=99, srv473869728=72, srv1233361248=10, srv294577225=66, srv1776623870=43, srv1786134674=44, srv1760688380=40, srv1555052259=30, srv526496352=75, srv988834822=98, srv593474482=77, srv137739713=18, srv1732205895=37, srv1436216877=23, srv1422069607=22, srv739393629=83, srv1309142990=15, srv1156746682=7, srv1792577933=45, srv1468006260=25, srv305857995=67, srv1878613400=51, srv974599524=97, srv1329761749=17, srv1863862330=50, srv2127502907=61, srv791712926=87, srv71737410=81, srv122406504=9, srv472097156=71, srv854413237=91, srv1080071534=4, srv1994955403=58, srv1321796152=16, srv1257968344=12, srv1570910827=32, srv2040580910=59, srv652705982=79, srv923154683=93, srv633057988=78, srv1483265914=29, srv1958797122=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 05:53:13,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 05:53:13,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 05:53:13,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 05:53:13,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 05:53:13,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 05:53:13,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-01 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-01 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-01 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-01 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-01 05:53:13,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-01 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-01 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-01 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-01 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-01 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-01 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-01 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-01 05:53:13,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-01 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-01 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-01 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-01 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-01 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-01 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-01 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-01 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-01 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-01 05:53:13,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-01 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-01 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-01 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-01 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-01 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-01 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-01 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-01 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-01 05:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-01 05:53:13,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-01 05:53:13,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-01 05:53:13,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-01 05:53:13,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-01 05:53:13,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-01 05:53:13,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-01 05:53:13,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-01 05:53:13,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-01 05:53:13,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-01 05:53:13,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-01 05:53:13,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-01 05:53:13,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-01 05:53:13,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-01 05:53:13,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-01 05:53:13,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-01 05:53:13,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-01 05:53:13,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-01 05:53:13,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-01 05:53:13,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-01 05:53:13,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-01 05:53:13,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-01 05:53:13,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-01 05:53:13,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-01 05:53:13,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-01 05:53:13,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-01 05:53:13,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-01 05:53:13,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-01 05:53:13,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-01 05:53:13,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-01 05:53:13,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-01 05:53:13,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-01 05:53:13,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-01 05:53:13,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-01 05:53:13,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-01 05:53:13,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-01 05:53:13,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-01 05:53:13,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-01 05:53:13,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-01 05:53:13,446 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03949992936074289, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05731555593152367, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03949992936074289, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17191666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:53:13,446 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-01 05:53:13,447 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17617469779378653, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03949992936074289, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05731555593152367, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03949992936074289, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17191666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-01 05:53:13,712 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 381 ms to try 5000 different iterations. Found a solution that moves 2553 regions; Going from a computed imbalance of 0.17617469779378653 to a new imbalance of 0.013987843292273846. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021501790005441047); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032143756779315115, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2836666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021501790005441047); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.03908333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:53:13,723 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2127502907:78 , srv1386855364:79 , srv1665609579:79 , srv180321322:79 , srv342051211:79 , srv130096813:80 , srv43977079:80 , srv1878613400:81 , srv288693666:81 , srv1253192618:82 , srv1436216877:82 , srv1671713106:82 , srv852894486:82 , srv854413237:82 , srv1592016503:83 , srv1760688380:83 , srv2040580910:83 , srv684963208:83 , srv750359641:83 , srv966526417:83 , srv974599524:83 , srv1049335747:84 , srv122406504:84 , srv1562800769:84 , srv1792577933:84 , srv278425528:84 , srv526496352:84 , srv589573107:84 , srv71737410:84 , srv1104237660:85 , srv1233361248:85 , srv1275523832:85 , srv1382502663:85 , srv1555052259:85 , srv1994955403:85 , srv2118469471:85 , srv294577225:85 , srv784388946:85 , srv791712926:85 , srv91646931:85 , srv956716333:85 , srv1061972837:86 , srv1109218513:86 , srv1156746682:86 , srv137739713:86 , srv1422069607:86 , srv1468006260:86 , srv1478126384:86 , srv1744236186:86 , srv1762938620:86 , srv1776623870:86 , srv1826652184:86 , srv473869728:86 , srv593474482:86 , srv652705982:86 , srv739393629:86 , srv838736206:86 , srv923154683:86 , srv1018510925:87 , srv1080071534:87 , srv1200921403:87 , srv1321796152:87 , srv1329761749:87 , srv1467513298:87 , srv1570910827:87 , srv1732205895:87 , srv1752418177:87 , srv1814763691:87 , srv1885864467:87 , srv1975989725:87 , srv1991516990:87 , srv305857995:87 , srv472097156:87 , srv526204737:87 , srv633057988:87 , srv945944631:87 , srv994459734:87 , srv1309142990:88 , srv1472282925:88 , srv1477582655:88 , srv1483265914:88 , srv1726792032:88 , srv1841293332:88 , srv1930816390:88 , srv286482742:88 , srv727363266:88 , srv1257968344:89 , srv1786134674:89 , srv1863862330:89 , srv476812482:89 , srv795682722:89 , srv1026963014:90 , srv1408770410:90 , srv1888637889:90 , srv223074031:90 , srv1958797122:91 , srv375431640:91 , srv782755101:91 , srv1764140497:92 , srv988834822:524 } 2024-02-01 05:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 05:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1930816390=54, srv2118469471=60, srv1018510925=0, srv1975989725=56, srv795682722=88, srv1049335747=2, srv684963208=80, srv838736206=89, srv375431640=69, srv1478126384=28, srv1109218513=6, srv1200921403=8, srv288693666=65, srv1061972837=3, srv852894486=90, srv1762938620=41, srv1888637889=53, srv956716333=95, srv1253192618=11, srv1472282925=26, srv784388946=86, srv1885864467=52, srv180321322=46, srv1991516990=57, srv1104237660=5, srv43977079=70, srv286482742=64, srv1467513298=24, srv750359641=84, srv1826652184=48, srv223074031=62, srv476812482=73, srv1562800769=31, srv1386855364=20, srv1744236186=38, srv1026963014=1, srv589573107=76, srv727363266=82, srv1592016503=33, srv782755101=85, srv130096813=14, srv342051211=68, srv91646931=92, srv1726792032=36, srv1841293332=49, srv1814763691=47, srv1275523832=13, srv966526417=96, srv1665609579=34, srv1408770410=21, srv1671713106=35, srv945944631=94, srv526204737=74, srv1477582655=27, srv278425528=63, srv1752418177=39, srv1382502663=19, srv1764140497=42, srv994459734=99, srv473869728=72, srv1233361248=10, srv294577225=66, srv1776623870=43, srv1786134674=44, srv1760688380=40, srv1555052259=30, srv526496352=75, srv988834822=98, srv593474482=77, srv137739713=18, srv1732205895=37, srv1436216877=23, srv1422069607=22, srv739393629=83, srv1309142990=15, srv1156746682=7, srv1792577933=45, srv1468006260=25, srv305857995=67, srv1878613400=51, srv974599524=97, srv1329761749=17, srv1863862330=50, srv2127502907=61, srv791712926=87, srv71737410=81, srv122406504=9, srv472097156=71, srv854413237=91, srv1080071534=4, srv1994955403=58, srv1321796152=16, srv1257968344=12, srv1570910827=32, srv2040580910=59, srv652705982=79, srv923154683=93, srv633057988=78, srv1483265914=29, srv1958797122=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 05:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 05:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 05:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 05:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 05:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 05:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 05:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 05:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 05:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 05:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 05:53:13,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 05:53:13,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-01 05:53:13,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-01 05:53:13,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-01 05:53:13,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-01 05:53:13,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-01 05:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-01 05:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-01 05:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-01 05:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-01 05:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-01 05:53:13,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-01 05:53:13,863 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021501790005441047); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032143756779315115, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021501790005441047); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.03908333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:53:13,863 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-01 05:53:13,863 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013969966804192005, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021501790005441047); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032143756779315115, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021501790005441047); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.03908333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-01 05:53:14,101 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 332 ms to try 5000 different iterations. Found a solution that moves 977 regions; Going from a computed imbalance of 0.013969966804192005 to a new imbalance of 6.319641428313483E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005994968168977276); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009125635134638571); MoveCostFunction : (multiplier=7.0, imbalance=0.10855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005994968168977276); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:53:14,103 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1764140497:81 , srv1958797122:81 , srv782755101:81 , srv795682722:81 , srv1386855364:83 , srv1826652184:83 , srv375431640:83 , srv476812482:83 , srv1026963014:84 , srv1382502663:84 , srv1408770410:84 , srv1436216877:84 , srv1752418177:84 , srv1841293332:84 , srv1885864467:84 , srv278425528:84 , srv288693666:84 , srv305857995:84 , srv473869728:84 , srv956716333:84 , srv1018510925:85 , srv1049335747:85 , srv1104237660:85 , srv1156746682:85 , srv1257968344:85 , srv1309142990:85 , srv1321796152:85 , srv1422069607:85 , srv1472282925:85 , srv1477582655:85 , srv1483265914:85 , srv1555052259:85 , srv1562800769:85 , srv1570910827:85 , srv1726792032:85 , srv1744236186:85 , srv1786134674:85 , srv1792577933:85 , srv1888637889:85 , srv1930816390:85 , srv1994955403:85 , srv2127502907:85 , srv223074031:85 , srv286482742:85 , srv526204737:85 , srv526496352:85 , srv593474482:85 , srv652705982:85 , srv71737410:85 , srv727363266:85 , srv739393629:85 , srv750359641:85 , srv784388946:85 , srv966526417:85 , srv994459734:85 , srv1061972837:86 , srv1109218513:86 , srv1233361248:86 , srv130096813:86 , srv1329761749:86 , srv137739713:86 , srv1478126384:86 , srv1592016503:86 , srv1760688380:86 , srv1776623870:86 , srv1814763691:86 , srv1863862330:86 , srv1991516990:86 , srv43977079:86 , srv472097156:86 , srv589573107:86 , srv633057988:86 , srv838736206:86 , srv854413237:86 , srv923154683:86 , srv945944631:86 , srv1080071534:87 , srv1200921403:87 , srv122406504:87 , srv1467513298:87 , srv1468006260:87 , srv1665609579:87 , srv1671713106:87 , srv1732205895:87 , srv1762938620:87 , srv1878613400:87 , srv1975989725:87 , srv342051211:87 , srv684963208:87 , srv852894486:87 , srv91646931:87 , srv974599524:87 , srv1253192618:88 , srv1275523832:88 , srv180321322:88 , srv2040580910:88 , srv2118469471:88 , srv294577225:88 , srv791712926:89 , srv988834822:546 } 2024-02-01 05:53:14,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 05:53:14,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1930816390=54, srv2118469471=60, srv1018510925=0, srv1975989725=56, srv795682722=88, srv1049335747=2, srv684963208=80, srv838736206=89, srv375431640=69, srv1478126384=28, srv1109218513=6, srv1200921403=8, srv288693666=65, srv1061972837=3, srv852894486=90, srv1762938620=41, srv1888637889=53, srv956716333=95, srv1253192618=11, srv1472282925=26, srv784388946=86, srv1885864467=52, srv180321322=46, srv1991516990=57, srv1104237660=5, srv43977079=70, srv286482742=64, srv1467513298=24, srv750359641=84, srv1826652184=48, srv223074031=62, srv476812482=73, srv1562800769=31, srv1386855364=20, srv1744236186=38, srv1026963014=1, srv589573107=76, srv727363266=82, srv1592016503=33, srv782755101=85, srv130096813=14, srv342051211=68, srv91646931=92, srv1726792032=36, srv1841293332=49, srv1814763691=47, srv1275523832=13, srv966526417=96, srv1665609579=34, srv1408770410=21, srv1671713106=35, srv945944631=94, srv526204737=74, srv1477582655=27, srv278425528=63, srv1752418177=39, srv1382502663=19, srv1764140497=42, srv994459734=99, srv473869728=72, srv1233361248=10, srv294577225=66, srv1776623870=43, srv1786134674=44, srv1760688380=40, srv1555052259=30, srv526496352=75, srv988834822=98, srv593474482=77, srv137739713=18, srv1732205895=37, srv1436216877=23, srv1422069607=22, srv739393629=83, srv1309142990=15, srv1156746682=7, srv1792577933=45, srv1468006260=25, srv305857995=67, srv1878613400=51, srv974599524=97, srv1329761749=17, srv1863862330=50, srv2127502907=61, srv791712926=87, srv71737410=81, srv122406504=9, srv472097156=71, srv854413237=91, srv1080071534=4, srv1994955403=58, srv1321796152=16, srv1257968344=12, srv1570910827=32, srv2040580910=59, srv652705982=79, srv923154683=93, srv633057988=78, srv1483265914=29, srv1958797122=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 05:53:14,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 05:53:14,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 05:53:14,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 05:53:14,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 05:53:14,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 05:53:14,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 05:53:14,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 05:53:14,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 05:53:14,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 05:53:14,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 05:53:14,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 05:53:14,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 05:53:14,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 05:53:14,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 05:53:14,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 05:53:14,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 05:53:14,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 05:53:14,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 05:53:14,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 05:53:14,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 05:53:14,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 05:53:14,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 05:53:14,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 05:53:14,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 05:53:14,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 05:53:14,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 05:53:14,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 05:53:14,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-01 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-01 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-01 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-01 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-01 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-01 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-01 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-01 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-01 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-01 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-01 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-01 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-01 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-01 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-01 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-01 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-01 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-01 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-01 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-01 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-01 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-01 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-01 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-01 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-01 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-01 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-01 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-01 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-01 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-01 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-01 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-01 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-01 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-01 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-01 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-01 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-01 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-01 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-01 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-01 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-01 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-01 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-01 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-01 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-01 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-01 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-01 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-01 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-01 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-01 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-01 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-01 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-01 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-01 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-01 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-01 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-01 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-01 05:53:14,249 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005994968168977276); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009125635134638571); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005994968168977276); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:53:14,249 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-01 05:53:14,249 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.251230426135815E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005994968168977276); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009125635134638571); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005994968168977276); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-01 05:53:14,603 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 461 ms to try 5000 different iterations. Found a solution that moves 297 regions; Going from a computed imbalance of 6.251230426135815E-4 to a new imbalance of 4.641967651203914E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002551379987942442); MoveCostFunction : (multiplier=7.0, imbalance=0.033, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:53:14,604 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1275523832:82 , srv1026963014:83 , srv1049335747:83 , srv223074031:83 , srv342051211:83 , srv854413237:83 , srv945944631:83 , srv966526417:83 , srv1156746682:84 , srv1200921403:84 , srv1233361248:84 , srv1257968344:84 , srv130096813:84 , srv1329761749:84 , srv137739713:84 , srv1408770410:84 , srv1468006260:84 , srv1478126384:84 , srv1483265914:84 , srv1665609579:84 , srv1671713106:84 , srv1732205895:84 , srv1762938620:84 , srv1786134674:84 , srv1878613400:84 , srv1888637889:84 , srv1930816390:84 , srv1975989725:84 , srv1991516990:84 , srv1994955403:84 , srv2040580910:84 , srv2118469471:84 , srv2127502907:84 , srv286482742:84 , srv294577225:84 , srv476812482:84 , srv526204737:84 , srv593474482:84 , srv684963208:84 , srv71737410:84 , srv727363266:84 , srv791712926:84 , srv923154683:84 , srv994459734:84 , srv1018510925:85 , srv1061972837:85 , srv1080071534:85 , srv1104237660:85 , srv1109218513:85 , srv122406504:85 , srv1253192618:85 , srv1309142990:85 , srv1321796152:85 , srv1422069607:85 , srv1436216877:85 , srv1467513298:85 , srv1472282925:85 , srv1555052259:85 , srv1562800769:85 , srv1570910827:85 , srv1592016503:85 , srv1726792032:85 , srv1744236186:85 , srv1752418177:85 , srv1760688380:85 , srv1764140497:85 , srv1776623870:85 , srv1792577933:85 , srv180321322:85 , srv1826652184:85 , srv1841293332:85 , srv1863862330:85 , srv1885864467:85 , srv278425528:85 , srv288693666:85 , srv305857995:85 , srv375431640:85 , srv472097156:85 , srv473869728:85 , srv526496352:85 , srv589573107:85 , srv633057988:85 , srv652705982:85 , srv750359641:85 , srv782755101:85 , srv784388946:85 , srv795682722:85 , srv838736206:85 , srv852894486:85 , srv91646931:85 , srv956716333:85 , srv974599524:85 , srv1382502663:86 , srv1386855364:86 , srv1477582655:86 , srv1814763691:86 , srv1958797122:86 , srv43977079:86 , srv739393629:86 , srv988834822:631 } 2024-02-01 05:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 05:53:14,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1930816390=54, srv2118469471=60, srv1018510925=0, srv1975989725=56, srv795682722=88, srv1049335747=2, srv684963208=80, srv838736206=89, srv375431640=69, srv1478126384=28, srv1109218513=6, srv1200921403=8, srv288693666=65, srv1061972837=3, srv852894486=90, srv1762938620=41, srv1888637889=53, srv956716333=95, srv1253192618=11, srv1472282925=26, srv784388946=86, srv1885864467=52, srv180321322=46, srv1991516990=57, srv1104237660=5, srv43977079=70, srv286482742=64, srv1467513298=24, srv750359641=84, srv1826652184=48, srv223074031=62, srv476812482=73, srv1562800769=31, srv1386855364=20, srv1744236186=38, srv1026963014=1, srv589573107=76, srv727363266=82, srv1592016503=33, srv782755101=85, srv130096813=14, srv342051211=68, srv91646931=92, srv1726792032=36, srv1841293332=49, srv1814763691=47, srv1275523832=13, srv966526417=96, srv1665609579=34, srv1408770410=21, srv1671713106=35, srv945944631=94, srv526204737=74, srv1477582655=27, srv278425528=63, srv1752418177=39, srv1382502663=19, srv1764140497=42, srv994459734=99, srv473869728=72, srv1233361248=10, srv294577225=66, srv1776623870=43, srv1786134674=44, srv1760688380=40, srv1555052259=30, srv526496352=75, srv988834822=98, srv593474482=77, srv137739713=18, srv1732205895=37, srv1436216877=23, srv1422069607=22, srv739393629=83, srv1309142990=15, srv1156746682=7, srv1792577933=45, srv1468006260=25, srv305857995=67, srv1878613400=51, srv974599524=97, srv1329761749=17, srv1863862330=50, srv2127502907=61, srv791712926=87, srv71737410=81, srv122406504=9, srv472097156=71, srv854413237=91, srv1080071534=4, srv1994955403=58, srv1321796152=16, srv1257968344=12, srv1570910827=32, srv2040580910=59, srv652705982=79, srv923154683=93, srv633057988=78, srv1483265914=29, srv1958797122=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 05:53:14,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 05:53:14,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 05:53:14,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 05:53:14,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 05:53:14,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 05:53:14,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 05:53:14,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 05:53:14,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 05:53:14,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 05:53:14,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 05:53:14,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 05:53:14,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 05:53:14,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 05:53:14,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 05:53:14,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 05:53:14,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 05:53:14,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 05:53:14,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 05:53:14,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 05:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 05:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 05:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 05:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 05:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 05:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 05:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 05:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 05:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 05:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 05:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 05:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-01 05:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-01 05:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-01 05:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-01 05:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-01 05:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-01 05:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-01 05:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-01 05:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-01 05:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-01 05:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-01 05:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-01 05:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-01 05:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-01 05:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-01 05:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-01 05:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-01 05:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-01 05:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-01 05:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-01 05:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-01 05:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-01 05:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-01 05:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-01 05:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-01 05:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-01 05:53:14,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-01 05:53:14,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-01 05:53:14,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-01 05:53:14,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-01 05:53:14,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-01 05:53:14,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-01 05:53:14,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-01 05:53:14,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-01 05:53:14,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-01 05:53:14,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-01 05:53:14,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-01 05:53:14,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-01 05:53:14,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-01 05:53:14,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-01 05:53:14,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-01 05:53:14,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-01 05:53:14,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-01 05:53:14,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-01 05:53:14,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-01 05:53:14,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-01 05:53:14,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-01 05:53:14,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-01 05:53:14,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-01 05:53:14,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-01 05:53:14,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-01 05:53:14,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-01 05:53:14,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-01 05:53:14,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-01 05:53:14,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-01 05:53:14,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-01 05:53:14,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-01 05:53:14,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-01 05:53:14,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-01 05:53:14,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-01 05:53:14,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-01 05:53:14,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-01 05:53:14,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-01 05:53:14,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-01 05:53:14,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-01 05:53:14,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-01 05:53:14,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-01 05:53:14,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-01 05:53:14,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-01 05:53:14,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-01 05:53:14,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-01 05:53:14,787 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002551379987942442); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:53:14,787 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-01 05:53:14,788 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.43400380630353E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002551379987942442); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-01 05:53:15,026 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 373 ms to try 5000 different iterations. Found a solution that moves 46 regions; Going from a computed imbalance of 4.43400380630353E-5 to a new imbalance of 8.301789001398086E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.005111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:53:15,027 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1018510925:83 , srv1109218513:83 , srv1386855364:83 , srv1436216877:83 , srv1477582655:83 , srv1478126384:83 , srv1814763691:83 , srv1885864467:83 , srv1958797122:83 , srv305857995:83 , srv633057988:83 , srv739393629:83 , srv750359641:83 , srv91646931:83 , srv1061972837:84 , srv1080071534:84 , srv1104237660:84 , srv1156746682:84 , srv1200921403:84 , srv122406504:84 , srv1233361248:84 , srv1253192618:84 , srv1257968344:84 , srv1275523832:84 , srv1309142990:84 , srv1321796152:84 , srv137739713:84 , srv1382502663:84 , srv1408770410:84 , srv1422069607:84 , srv1467513298:84 , srv1472282925:84 , srv1483265914:84 , srv1555052259:84 , srv1562800769:84 , srv1570910827:84 , srv1592016503:84 , srv1671713106:84 , srv1726792032:84 , srv1732205895:84 , srv1744236186:84 , srv1752418177:84 , srv1760688380:84 , srv1764140497:84 , srv1776623870:84 , srv1786134674:84 , srv1792577933:84 , srv180321322:84 , srv1826652184:84 , srv1841293332:84 , srv1863862330:84 , srv1878613400:84 , srv1888637889:84 , srv1975989725:84 , srv1991516990:84 , srv2040580910:84 , srv2118469471:84 , srv2127502907:84 , srv278425528:84 , srv286482742:84 , srv288693666:84 , srv294577225:84 , srv342051211:84 , srv375431640:84 , srv43977079:84 , srv472097156:84 , srv473869728:84 , srv526496352:84 , srv589573107:84 , srv593474482:84 , srv652705982:84 , srv684963208:84 , srv71737410:84 , srv727363266:84 , srv782755101:84 , srv784388946:84 , srv791712926:84 , srv795682722:84 , srv838736206:84 , srv852894486:84 , srv854413237:84 , srv923154683:84 , srv956716333:84 , srv966526417:84 , srv994459734:84 , srv1026963014:85 , srv1049335747:85 , srv130096813:85 , srv1329761749:85 , srv1468006260:85 , srv1665609579:85 , srv1762938620:85 , srv1930816390:85 , srv1994955403:85 , srv223074031:85 , srv476812482:85 , srv526204737:85 , srv945944631:85 , srv974599524:85 , srv988834822:684 } 2024-02-01 05:53:15,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 05:53:15,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1930816390=54, srv2118469471=60, srv1018510925=0, srv1975989725=56, srv795682722=88, srv1049335747=2, srv684963208=80, srv838736206=89, srv375431640=69, srv1478126384=28, srv1109218513=6, srv1200921403=8, srv288693666=65, srv1061972837=3, srv852894486=90, srv1762938620=41, srv1888637889=53, srv956716333=95, srv1253192618=11, srv1472282925=26, srv784388946=86, srv1885864467=52, srv180321322=46, srv1991516990=57, srv1104237660=5, srv43977079=70, srv286482742=64, srv1467513298=24, srv750359641=84, srv1826652184=48, srv223074031=62, srv476812482=73, srv1562800769=31, srv1386855364=20, srv1744236186=38, srv1026963014=1, srv589573107=76, srv727363266=82, srv1592016503=33, srv782755101=85, srv130096813=14, srv342051211=68, srv91646931=92, srv1726792032=36, srv1841293332=49, srv1814763691=47, srv1275523832=13, srv966526417=96, srv1665609579=34, srv1408770410=21, srv1671713106=35, srv945944631=94, srv526204737=74, srv1477582655=27, srv278425528=63, srv1752418177=39, srv1382502663=19, srv1764140497=42, srv994459734=99, srv473869728=72, srv1233361248=10, srv294577225=66, srv1776623870=43, srv1786134674=44, srv1760688380=40, srv1555052259=30, srv526496352=75, srv988834822=98, srv593474482=77, srv137739713=18, srv1732205895=37, srv1436216877=23, srv1422069607=22, srv739393629=83, srv1309142990=15, srv1156746682=7, srv1792577933=45, srv1468006260=25, srv305857995=67, srv1878613400=51, srv974599524=97, srv1329761749=17, srv1863862330=50, srv2127502907=61, srv791712926=87, srv71737410=81, srv122406504=9, srv472097156=71, srv854413237=91, srv1080071534=4, srv1994955403=58, srv1321796152=16, srv1257968344=12, srv1570910827=32, srv2040580910=59, srv652705982=79, srv923154683=93, srv633057988=78, srv1483265914=29, srv1958797122=55} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 05:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 05:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 05:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 05:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 05:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 05:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 05:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 05:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 05:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 05:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 05:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 05:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 05:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 05:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 05:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 05:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 05:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 05:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 05:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 05:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 05:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 05:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 05:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 05:53:15,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 05:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 05:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 05:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 05:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 05:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 05:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 05:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-01 05:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-01 05:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-01 05:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-01 05:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-01 05:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-01 05:53:15,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-01 05:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-01 05:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-01 05:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-01 05:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-01 05:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-01 05:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-01 05:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-01 05:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-01 05:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-01 05:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-01 05:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-01 05:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-01 05:53:15,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-01 05:53:15,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-01 05:53:15,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-01 05:53:15,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-01 05:53:15,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-01 05:53:15,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-01 05:53:15,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-01 05:53:15,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-01 05:53:15,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-01 05:53:15,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-01 05:53:15,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-01 05:53:15,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-01 05:53:15,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-01 05:53:15,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-01 05:53:15,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-01 05:53:15,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-01 05:53:15,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-01 05:53:15,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-01 05:53:15,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-01 05:53:15,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-01 05:53:15,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-01 05:53:15,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-01 05:53:15,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-01 05:53:15,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-01 05:53:15,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-01 05:53:15,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-01 05:53:15,195 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.979690117040589E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:53:15,195 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1018510925:83 , srv1109218513:83 , srv1386855364:83 , srv1436216877:83 , srv1477582655:83 , srv1478126384:83 , srv1814763691:83 , srv1885864467:83 , srv1958797122:83 , srv305857995:83 , srv633057988:83 , srv739393629:83 , srv750359641:83 , srv91646931:83 , srv1061972837:84 , srv1080071534:84 , srv1104237660:84 , srv1156746682:84 , srv1200921403:84 , srv122406504:84 , srv1233361248:84 , srv1253192618:84 , srv1257968344:84 , srv1275523832:84 , srv1309142990:84 , srv1321796152:84 , srv137739713:84 , srv1382502663:84 , srv1408770410:84 , srv1422069607:84 , srv1467513298:84 , srv1472282925:84 , srv1483265914:84 , srv1555052259:84 , srv1562800769:84 , srv1570910827:84 , srv1592016503:84 , srv1671713106:84 , srv1726792032:84 , srv1732205895:84 , srv1744236186:84 , srv1752418177:84 , srv1760688380:84 , srv1764140497:84 , srv1776623870:84 , srv1786134674:84 , srv1792577933:84 , srv180321322:84 , srv1826652184:84 , srv1841293332:84 , srv1863862330:84 , srv1878613400:84 , srv1888637889:84 , srv1975989725:84 , srv1991516990:84 , srv2040580910:84 , srv2118469471:84 , srv2127502907:84 , srv278425528:84 , srv286482742:84 , srv288693666:84 , srv294577225:84 , srv342051211:84 , srv375431640:84 , srv43977079:84 , srv472097156:84 , srv473869728:84 , srv526496352:84 , srv589573107:84 , srv593474482:84 , srv652705982:84 , srv684963208:84 , srv71737410:84 , srv727363266:84 , srv782755101:84 , srv784388946:84 , srv791712926:84 , srv795682722:84 , srv838736206:84 , srv852894486:84 , srv854413237:84 , srv923154683:84 , srv956716333:84 , srv966526417:84 , srv994459734:84 , srv1026963014:85 , srv1049335747:85 , srv130096813:85 , srv1329761749:85 , srv1468006260:85 , srv1665609579:85 , srv1762938620:85 , srv1930816390:85 , srv1994955403:85 , srv223074031:85 , srv476812482:85 , srv526204737:85 , srv945944631:85 , srv974599524:85 , srv988834822:684 } 2024-02-01 05:53:15,218 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=347 (was 333) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=2283 (was 3151) 2024-02-01 05:53:15,227 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=347, ProcessCount=167, AvailableMemoryMB=2282 2024-02-01 05:53:15,227 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-01 05:53:15,228 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-02-01 05:53:15,238 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1050260527:84 , srv1066102363:84 , srv1155913210:84 , srv1174027578:84 , srv1225031904:84 , srv1298058476:84 , srv1312943030:84 , srv1392370905:84 , srv1544310486:84 , srv1617428725:84 , srv1722746256:84 , srv175001740:84 , srv1887422625:84 , srv1917200888:84 , srv1952303231:84 , srv2042105893:84 , srv234074253:84 , srv298604318:84 , srv336690077:84 , srv349541778:84 , srv421427548:84 , srv44077860:84 , srv607302309:84 , srv698323462:84 , srv699961452:84 , srv73568141:84 , srv81673430:84 , srv931658153:84 , srv965631011:84 , srv968145969:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-02-01 05:53:15,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 05:53:15,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1917200888=13, srv349541778=19, srv1174027578=3, srv1617428725=9, srv931658153=27, srv968145969=29, srv1225031904=4, srv1050260527=0, srv336690077=18, srv1952303231=14, srv2042105893=15, srv1312943030=6, srv73568141=25, srv1544310486=8, srv1887422625=12, srv234074253=16, srv1066102363=1, srv175001740=11, srv44077860=21, srv298604318=17, srv1155913210=2, srv1298058476=5, srv607302309=22, srv965631011=28, srv1722746256=10, srv421427548=20, srv698323462=23, srv699961452=24, srv1392370905=7, srv81673430=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 05:53:15,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 05:53:15,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 05:53:15,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 05:53:15,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 05:53:15,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 05:53:15,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 05:53:15,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 05:53:15,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 05:53:15,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 05:53:15,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-01 05:53:15,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.136983557838848, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:53:15,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912624673982104, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.136983557838848, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-02-01 05:55:15,250 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6942280 different iterations. Found a solution that moves 1846 regions; Going from a computed imbalance of 0.9912624673982104 to a new imbalance of 4.308656090753194E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6837037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:55:15,255 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1050260527:90 , srv1066102363:90 , srv1155913210:90 , srv1174027578:90 , srv1225031904:90 , srv1298058476:90 , srv1312943030:90 , srv1392370905:90 , srv1544310486:90 , srv1617428725:90 , srv1722746256:90 , srv175001740:90 , srv1887422625:90 , srv1917200888:90 , srv1952303231:90 , srv2042105893:90 , srv234074253:90 , srv298604318:90 , srv336690077:90 , srv349541778:90 , srv421427548:90 , srv44077860:90 , srv607302309:90 , srv698323462:90 , srv699961452:90 , srv73568141:90 , srv81673430:90 , srv931658153:90 , srv965631011:90 , srv968145969:90 } 2024-02-01 05:55:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 05:55:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1917200888=13, srv349541778=19, srv1174027578=3, srv1617428725=9, srv931658153=27, srv968145969=29, srv1225031904=4, srv1050260527=0, srv336690077=18, srv1952303231=14, srv2042105893=15, srv1312943030=6, srv73568141=25, srv1544310486=8, srv1887422625=12, srv234074253=16, srv1066102363=1, srv175001740=11, srv44077860=21, srv298604318=17, srv1155913210=2, srv1298058476=5, srv607302309=22, srv965631011=28, srv1722746256=10, srv421427548=20, srv698323462=23, srv699961452=24, srv1392370905=7, srv81673430=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 05:55:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 05:55:15,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 05:55:15,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 05:55:15,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 05:55:15,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 05:55:15,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 05:55:15,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 05:55:15,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 05:55:15,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 05:55:15,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-01 05:55:15,272 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 05:55:15,272 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1050260527:90 , srv1066102363:90 , srv1155913210:90 , srv1174027578:90 , srv1225031904:90 , srv1298058476:90 , srv1312943030:90 , srv1392370905:90 , srv1544310486:90 , srv1617428725:90 , srv1722746256:90 , srv175001740:90 , srv1887422625:90 , srv1917200888:90 , srv1952303231:90 , srv2042105893:90 , srv234074253:90 , srv298604318:90 , srv336690077:90 , srv349541778:90 , srv421427548:90 , srv44077860:90 , srv607302309:90 , srv698323462:90 , srv699961452:90 , srv73568141:90 , srv81673430:90 , srv931658153:90 , srv965631011:90 , srv968145969:90 } 2024-02-01 05:55:15,285 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=235 (was 347), ProcessCount=167 (was 167), AvailableMemoryMB=2445 (was 2282) - AvailableMemoryMB LEAK? -