2023-11-11 20:55:17,351 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-11-11 20:55:17,630 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-11-11 20:55:17,727 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-11-11 20:55:17,783 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-11 20:55:17,789 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-11-11 20:55:17,802 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=97, ProcessCount=170, AvailableMemoryMB=1636 2023-11-11 20:55:17,804 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-11 20:55:17,804 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-11-11 20:55:17,913 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1006958827:84 , srv1072940955:84 , srv1093856147:84 , srv1094078004:84 , srv1094681221:84 , srv1122970211:84 , srv1125648634:84 , srv1131673350:84 , srv1145038612:84 , srv1160797971:84 , srv1192695892:84 , srv1204458117:84 , srv1212661309:84 , srv1218202541:84 , srv1229327611:84 , srv1241730410:84 , srv1259325391:84 , srv1273148539:84 , srv1286374882:84 , srv1338881444:84 , srv1342325034:84 , srv1354731284:84 , srv1391786565:84 , srv1437409308:84 , srv1453892715:84 , srv1456905324:84 , srv1467222976:84 , srv1483704742:84 , srv1501336588:84 , srv1531753662:84 , srv1555820673:84 , srv1566826040:84 , srv1583796260:84 , srv1598032065:84 , srv1642454058:84 , srv1716613309:84 , srv1743657961:84 , srv1772520018:84 , srv1780760124:84 , srv18038785:84 , srv1837853444:84 , srv1874430857:84 , srv1903885467:84 , srv1911169586:84 , srv1943725808:84 , srv1953285135:84 , srv1954578515:84 , srv1966798838:84 , srv1971337060:84 , srv203102642:84 , srv2043427907:84 , srv2053071672:84 , srv2064207842:84 , srv2085386205:84 , srv2110423993:84 , srv2114679593:84 , srv226169559:84 , srv285283751:84 , srv343780200:84 , srv375695875:84 , srv389978366:84 , srv404318557:84 , srv40819841:84 , srv4259515:84 , srv42889655:84 , srv431160466:84 , srv440864950:84 , srv46456461:84 , srv478672610:84 , srv48227143:84 , srv531140638:84 , srv539361443:84 , srv548454019:84 , srv604466476:84 , srv617308682:84 , srv625194687:84 , srv650145184:84 , srv670752142:84 , srv68125779:84 , srv691672014:84 , srv691803755:84 , srv740730680:84 , srv743582204:84 , srv753533405:84 , srv781615027:84 , srv790158314:84 , srv800356455:84 , srv810700249:84 , srv828045274:84 , srv873794342:84 , srv902669367:84 , srv904877743:84 , srv906312380:84 , srv914552302:84 , srv928570216:84 , srv940066849:84 , srv970588901:84 , srv97194566:84 , srv980570543:84 , srv883329382:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-11-11 20:55:17,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 20:55:17,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv753533405=83, srv1874430857=41, srv1531753662=29, srv1743657961=36, srv1943725808=44, srv1555820673=30, srv42889655=64, srv1072940955=1, srv625194687=75, srv2114679593=55, srv1642454058=34, srv539361443=71, srv740730680=81, srv1218202541=13, srv1501336588=28, srv1212661309=12, srv810700249=87, srv1093856147=2, srv883329382=90, srv4259515=63, srv691672014=79, srv1903885467=42, srv1354731284=21, srv670752142=77, srv1241730410=15, srv790158314=85, srv1273148539=17, srv2043427907=50, srv691803755=80, srv617308682=74, srv1971337060=48, srv1953285135=45, srv1006958827=0, srv904877743=92, srv1094681221=4, srv203102642=49, srv1467222976=26, srv604466476=73, srv389978366=60, srv1286374882=18, srv2110423993=54, srv1259325391=16, srv1094078004=3, srv478672610=68, srv800356455=86, srv40819841=62, srv828045274=88, srv1122970211=5, srv1229327611=14, srv928570216=95, srv68125779=78, srv440864950=66, srv902669367=91, srv431160466=65, srv1483704742=27, srv531140638=70, srv914552302=94, srv906312380=93, srv1837853444=40, srv743582204=82, srv1966798838=47, srv1456905324=25, srv1204458117=11, srv1911169586=43, srv980570543=99, srv1125648634=6, srv1583796260=32, srv1453892715=24, srv1391786565=22, srv781615027=84, srv548454019=72, srv970588901=97, srv1131673350=7, srv2053071672=51, srv1780760124=38, srv1598032065=33, srv97194566=98, srv1342325034=20, srv285283751=57, srv48227143=69, srv873794342=89, srv1716613309=35, srv226169559=56, srv18038785=39, srv1566826040=31, srv1954578515=46, srv1160797971=9, srv1338881444=19, srv2085386205=53, srv1192695892=10, srv404318557=61, srv2064207842=52, srv375695875=59, srv940066849=96, srv1145038612=8, srv1437409308=23, srv1772520018=37, srv46456461=67, srv343780200=58, srv650145184=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 20:55:17,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 20:55:17,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 20:55:17,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 20:55:17,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 20:55:17,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 20:55:17,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 20:55:17,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 20:55:17,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 20:55:17,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 20:55:17,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 20:55:17,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 20:55:17,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 20:55:17,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 20:55:17,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 20:55:17,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 20:55:17,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 20:55:17,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 20:55:17,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 20:55:17,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 20:55:17,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 20:55:17,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 20:55:17,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 20:55:17,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 20:55:17,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 20:55:17,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 20:55:17,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 20:55:17,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 20:55:17,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 20:55:17,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 20:55:17,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 20:55:17,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-11 20:55:17,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-11 20:55:17,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-11 20:55:17,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-11 20:55:18,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-11 20:55:18,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-11 20:55:18,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-11 20:55:18,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-11 20:55:18,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-11 20:55:18,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-11 20:55:18,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-11 20:55:18,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-11 20:55:18,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-11 20:55:18,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-11 20:55:18,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-11 20:55:18,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-11 20:55:18,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-11 20:55:18,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-11 20:55:18,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-11 20:55:18,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-11 20:55:18,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-11 20:55:18,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-11 20:55:18,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-11 20:55:18,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-11 20:55:18,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-11 20:55:18,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-11 20:55:18,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-11 20:55:18,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-11 20:55:18,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-11 20:55:18,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-11 20:55:18,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-11 20:55:18,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-11 20:55:18,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-11 20:55:18,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-11 20:55:18,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-11 20:55:18,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-11 20:55:18,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-11 20:55:18,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-11 20:55:18,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-11 20:55:18,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-11 20:55:18,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-11 20:55:18,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-11 20:55:18,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-11 20:55:18,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-11 20:55:18,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-11 20:55:18,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-11 20:55:18,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-11 20:55:18,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-11 20:55:18,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-11 20:55:18,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-11 20:55:18,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-11 20:55:18,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-11 20:55:18,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-11 20:55:18,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-11 20:55:18,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-11 20:55:18,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-11 20:55:18,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-11 20:55:18,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-11 20:55:18,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-11 20:55:18,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-11 20:55:18,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-11 20:55:18,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-11 20:55:18,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-11 20:55:18,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-11 20:55:18,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-11 20:55:18,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-11 20:55:18,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-11 20:55:18,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-11 20:55:18,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-11 20:55:18,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-11 20:55:18,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-11 20:55:18,280 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:55:18,280 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-11 20:55:18,281 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-11 20:55:18,669 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 639 ms to try 5000 different iterations. Found a solution that moves 4696 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17801157921203137. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04062886383632334, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05687326174253212, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5217777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04062886383632334, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17308333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:55:18,679 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv285283751:81 , srv1122970211:82 , srv1160797971:82 , srv1259325391:82 , srv2085386205:82 , srv753533405:82 , srv980570543:82 , srv1342325034:83 , srv1501336588:83 , srv1598032065:83 , srv1716613309:83 , srv203102642:83 , srv42889655:83 , srv904877743:83 , srv1072940955:84 , srv1354731284:84 , srv1743657961:84 , srv1772520018:84 , srv1954578515:84 , srv1966798838:84 , srv431160466:84 , srv48227143:84 , srv740730680:84 , srv902669367:84 , srv1212661309:85 , srv1391786565:85 , srv1456905324:85 , srv1642454058:85 , srv1780760124:85 , srv18038785:85 , srv1874430857:85 , srv226169559:85 , srv4259515:85 , srv46456461:85 , srv548454019:85 , srv650145184:85 , srv828045274:85 , srv873794342:85 , srv914552302:85 , srv97194566:85 , srv1131673350:86 , srv1273148539:86 , srv1286374882:86 , srv1566826040:86 , srv1911169586:86 , srv404318557:86 , srv440864950:86 , srv670752142:86 , srv68125779:86 , srv691672014:86 , srv781615027:86 , srv800356455:86 , srv810700249:86 , srv1094078004:87 , srv1094681221:87 , srv1338881444:87 , srv1467222976:87 , srv1483704742:87 , srv1531753662:87 , srv2053071672:87 , srv2064207842:87 , srv2110423993:87 , srv2114679593:87 , srv375695875:87 , srv604466476:87 , srv625194687:87 , srv928570216:87 , srv970588901:87 , srv1093856147:88 , srv1145038612:88 , srv1192695892:88 , srv1241730410:88 , srv1453892715:88 , srv1555820673:88 , srv1583796260:88 , srv1903885467:88 , srv1953285135:88 , srv531140638:88 , srv617308682:88 , srv691803755:88 , srv743582204:88 , srv790158314:88 , srv906312380:88 , srv1204458117:89 , srv1437409308:89 , srv1837853444:89 , srv1971337060:89 , srv478672610:89 , srv539361443:89 , srv1006958827:90 , srv1229327611:90 , srv940066849:90 , srv2043427907:91 , srv343780200:91 , srv40819841:91 , srv1125648634:92 , srv389978366:92 , srv1218202541:95 , srv1943725808:96 , srv883329382:451 } 2023-11-11 20:55:18,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 20:55:18,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv753533405=83, srv1874430857=41, srv1531753662=29, srv1743657961=36, srv1943725808=44, srv1555820673=30, srv42889655=64, srv1072940955=1, srv625194687=75, srv2114679593=55, srv1642454058=34, srv539361443=71, srv740730680=81, srv1218202541=13, srv1501336588=28, srv1212661309=12, srv810700249=87, srv1093856147=2, srv883329382=90, srv4259515=63, srv691672014=79, srv1903885467=42, srv1354731284=21, srv670752142=77, srv1241730410=15, srv790158314=85, srv1273148539=17, srv2043427907=50, srv691803755=80, srv617308682=74, srv1971337060=48, srv1953285135=45, srv1006958827=0, srv904877743=92, srv1094681221=4, srv203102642=49, srv1467222976=26, srv604466476=73, srv389978366=60, srv1286374882=18, srv2110423993=54, srv1259325391=16, srv1094078004=3, srv478672610=68, srv800356455=86, srv40819841=62, srv828045274=88, srv1122970211=5, srv1229327611=14, srv928570216=95, srv68125779=78, srv440864950=66, srv902669367=91, srv431160466=65, srv1483704742=27, srv531140638=70, srv914552302=94, srv906312380=93, srv1837853444=40, srv743582204=82, srv1966798838=47, srv1456905324=25, srv1204458117=11, srv1911169586=43, srv980570543=99, srv1125648634=6, srv1583796260=32, srv1453892715=24, srv1391786565=22, srv781615027=84, srv548454019=72, srv970588901=97, srv1131673350=7, srv2053071672=51, srv1780760124=38, srv1598032065=33, srv97194566=98, srv1342325034=20, srv285283751=57, srv48227143=69, srv873794342=89, srv1716613309=35, srv226169559=56, srv18038785=39, srv1566826040=31, srv1954578515=46, srv1160797971=9, srv1338881444=19, srv2085386205=53, srv1192695892=10, srv404318557=61, srv2064207842=52, srv375695875=59, srv940066849=96, srv1145038612=8, srv1437409308=23, srv1772520018=37, srv46456461=67, srv343780200=58, srv650145184=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 20:55:18,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 20:55:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 20:55:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 20:55:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 20:55:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 20:55:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 20:55:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 20:55:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 20:55:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 20:55:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 20:55:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 20:55:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 20:55:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 20:55:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-11 20:55:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-11 20:55:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-11 20:55:18,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-11 20:55:18,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-11 20:55:18,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-11 20:55:18,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-11 20:55:18,815 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04062886383632334, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05687326174253212, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04062886383632334, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17308333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:55:18,815 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-11 20:55:18,816 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17797869711722827, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04062886383632334, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05687326174253212, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04062886383632334, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17308333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-11 20:55:19,097 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 378 ms to try 5000 different iterations. Found a solution that moves 2660 regions; Going from a computed imbalance of 0.17797869711722827 to a new imbalance of 0.014867108084669586. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022175596428106778); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03658853015250187, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29555555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022175596428106778); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043583333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:55:19,107 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1218202541:76 , srv1125648634:77 , srv1943725808:79 , srv389978366:79 , srv2043427907:80 , srv940066849:81 , srv1229327611:82 , srv1273148539:82 , srv343780200:82 , srv478672610:82 , srv539361443:82 , srv790158314:82 , srv1094078004:83 , srv1192695892:83 , srv1204458117:83 , srv1437409308:83 , srv1837853444:83 , srv1971337060:83 , srv617308682:83 , srv1145038612:84 , srv1241730410:84 , srv1286374882:84 , srv1555820673:84 , srv40819841:84 , srv625194687:84 , srv691803755:84 , srv928570216:84 , srv1006958827:85 , srv1093856147:85 , srv1131673350:85 , srv1391786565:85 , srv1453892715:85 , srv1456905324:85 , srv1531753662:85 , srv1903885467:85 , srv1953285135:85 , srv2064207842:85 , srv2110423993:85 , srv375695875:85 , srv48227143:85 , srv531140638:85 , srv670752142:85 , srv68125779:85 , srv691672014:85 , srv743582204:85 , srv810700249:85 , srv906312380:85 , srv914552302:85 , srv97194566:85 , srv1338881444:86 , srv1483704742:86 , srv1583796260:86 , srv18038785:86 , srv1874430857:86 , srv203102642:86 , srv226169559:86 , srv404318557:86 , srv440864950:86 , srv781615027:86 , srv828045274:86 , srv970588901:86 , srv1094681221:87 , srv1354731284:87 , srv1467222976:87 , srv1566826040:87 , srv1642454058:87 , srv1743657961:87 , srv1911169586:87 , srv1954578515:87 , srv548454019:87 , srv604466476:87 , srv902669367:87 , srv1072940955:88 , srv1122970211:88 , srv1160797971:88 , srv1212661309:88 , srv1259325391:88 , srv1342325034:88 , srv1772520018:88 , srv1780760124:88 , srv2053071672:88 , srv431160466:88 , srv46456461:88 , srv800356455:88 , srv1501336588:89 , srv1716613309:89 , srv2114679593:89 , srv4259515:89 , srv42889655:89 , srv650145184:89 , srv1598032065:90 , srv1966798838:90 , srv2085386205:90 , srv740730680:90 , srv753533405:90 , srv285283751:91 , srv873794342:91 , srv904877743:91 , srv980570543:91 , srv883329382:520 } 2023-11-11 20:55:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 20:55:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv753533405=83, srv1874430857=41, srv1531753662=29, srv1743657961=36, srv1943725808=44, srv1555820673=30, srv42889655=64, srv1072940955=1, srv625194687=75, srv2114679593=55, srv1642454058=34, srv539361443=71, srv740730680=81, srv1218202541=13, srv1501336588=28, srv1212661309=12, srv810700249=87, srv1093856147=2, srv883329382=90, srv4259515=63, srv691672014=79, srv1903885467=42, srv1354731284=21, srv670752142=77, srv1241730410=15, srv790158314=85, srv1273148539=17, srv2043427907=50, srv691803755=80, srv617308682=74, srv1971337060=48, srv1953285135=45, srv1006958827=0, srv904877743=92, srv1094681221=4, srv203102642=49, srv1467222976=26, srv604466476=73, srv389978366=60, srv1286374882=18, srv2110423993=54, srv1259325391=16, srv1094078004=3, srv478672610=68, srv800356455=86, srv40819841=62, srv828045274=88, srv1122970211=5, srv1229327611=14, srv928570216=95, srv68125779=78, srv440864950=66, srv902669367=91, srv431160466=65, srv1483704742=27, srv531140638=70, srv914552302=94, srv906312380=93, srv1837853444=40, srv743582204=82, srv1966798838=47, srv1456905324=25, srv1204458117=11, srv1911169586=43, srv980570543=99, srv1125648634=6, srv1583796260=32, srv1453892715=24, srv1391786565=22, srv781615027=84, srv548454019=72, srv970588901=97, srv1131673350=7, srv2053071672=51, srv1780760124=38, srv1598032065=33, srv97194566=98, srv1342325034=20, srv285283751=57, srv48227143=69, srv873794342=89, srv1716613309=35, srv226169559=56, srv18038785=39, srv1566826040=31, srv1954578515=46, srv1160797971=9, srv1338881444=19, srv2085386205=53, srv1192695892=10, srv404318557=61, srv2064207842=52, srv375695875=59, srv940066849=96, srv1145038612=8, srv1437409308=23, srv1772520018=37, srv46456461=67, srv343780200=58, srv650145184=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 20:55:19,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 20:55:19,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 20:55:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 20:55:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 20:55:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 20:55:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 20:55:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 20:55:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 20:55:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 20:55:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 20:55:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 20:55:19,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 20:55:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 20:55:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 20:55:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 20:55:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 20:55:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 20:55:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 20:55:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 20:55:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 20:55:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 20:55:19,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 20:55:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 20:55:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 20:55:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 20:55:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 20:55:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 20:55:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 20:55:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 20:55:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 20:55:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-11 20:55:19,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-11 20:55:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-11 20:55:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-11 20:55:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-11 20:55:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-11 20:55:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-11 20:55:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-11 20:55:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-11 20:55:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-11 20:55:19,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-11 20:55:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-11 20:55:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-11 20:55:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-11 20:55:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-11 20:55:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-11 20:55:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-11 20:55:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-11 20:55:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-11 20:55:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-11 20:55:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-11 20:55:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-11 20:55:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-11 20:55:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-11 20:55:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-11 20:55:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-11 20:55:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-11 20:55:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-11 20:55:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-11 20:55:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-11 20:55:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-11 20:55:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-11 20:55:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-11 20:55:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-11 20:55:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-11 20:55:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-11 20:55:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-11 20:55:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-11 20:55:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-11 20:55:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-11 20:55:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-11 20:55:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-11 20:55:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-11 20:55:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-11 20:55:19,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-11 20:55:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-11 20:55:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-11 20:55:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-11 20:55:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-11 20:55:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-11 20:55:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-11 20:55:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-11 20:55:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-11 20:55:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-11 20:55:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-11 20:55:19,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-11 20:55:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-11 20:55:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-11 20:55:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-11 20:55:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-11 20:55:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-11 20:55:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-11 20:55:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-11 20:55:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-11 20:55:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-11 20:55:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-11 20:55:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-11 20:55:19,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-11 20:55:19,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-11 20:55:19,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-11 20:55:19,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-11 20:55:19,268 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022175596428106778); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03658853015250187, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022175596428106778); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043583333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:55:19,269 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-11 20:55:19,269 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01484848236657413, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022175596428106778); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03658853015250187, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022175596428106778); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043583333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-11 20:55:19,597 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 431 ms to try 5000 different iterations. Found a solution that moves 1015 regions; Going from a computed imbalance of 0.01484848236657413 to a new imbalance of 7.663375021908582E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005102759975884884); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008206099398622182); MoveCostFunction : (multiplier=7.0, imbalance=0.11277777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005102759975884884); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:55:19,600 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv873794342:81 , srv40819841:82 , srv1006958827:83 , srv1094681221:83 , srv1943725808:83 , srv2053071672:83 , srv2114679593:83 , srv604466476:83 , srv650145184:83 , srv740730680:83 , srv800356455:83 , srv904877743:83 , srv1093856147:84 , srv1338881444:84 , srv1453892715:84 , srv1467222976:84 , srv1583796260:84 , srv1598032065:84 , srv1780760124:84 , srv1903885467:84 , srv1911169586:84 , srv1953285135:84 , srv1966798838:84 , srv2110423993:84 , srv343780200:84 , srv4259515:84 , srv1212661309:85 , srv1229327611:85 , srv1241730410:85 , srv1437409308:85 , srv1483704742:85 , srv1642454058:85 , srv1772520018:85 , srv1837853444:85 , srv1971337060:85 , srv2043427907:85 , srv2064207842:85 , srv2085386205:85 , srv285283751:85 , srv42889655:85 , srv440864950:85 , srv46456461:85 , srv531140638:85 , srv548454019:85 , srv743582204:85 , srv970588901:85 , srv980570543:85 , srv1072940955:86 , srv1145038612:86 , srv1204458117:86 , srv1501336588:86 , srv1531753662:86 , srv1555820673:86 , srv1566826040:86 , srv1743657961:86 , srv18038785:86 , srv1954578515:86 , srv226169559:86 , srv375695875:86 , srv404318557:86 , srv431160466:86 , srv539361443:86 , srv617308682:86 , srv625194687:86 , srv670752142:86 , srv753533405:86 , srv781615027:86 , srv902669367:86 , srv1122970211:87 , srv1131673350:87 , srv1160797971:87 , srv1192695892:87 , srv1218202541:87 , srv1259325391:87 , srv1286374882:87 , srv1342325034:87 , srv1354731284:87 , srv1391786565:87 , srv1456905324:87 , srv1716613309:87 , srv1874430857:87 , srv389978366:87 , srv478672610:87 , srv68125779:87 , srv691672014:87 , srv691803755:87 , srv790158314:87 , srv810700249:87 , srv828045274:87 , srv906312380:87 , srv928570216:87 , srv940066849:87 , srv97194566:87 , srv1094078004:88 , srv914552302:88 , srv1125648634:89 , srv203102642:89 , srv48227143:89 , srv1273148539:90 , srv883329382:532 } 2023-11-11 20:55:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 20:55:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv753533405=83, srv1874430857=41, srv1531753662=29, srv1743657961=36, srv1943725808=44, srv1555820673=30, srv42889655=64, srv1072940955=1, srv625194687=75, srv2114679593=55, srv1642454058=34, srv539361443=71, srv740730680=81, srv1218202541=13, srv1501336588=28, srv1212661309=12, srv810700249=87, srv1093856147=2, srv883329382=90, srv4259515=63, srv691672014=79, srv1903885467=42, srv1354731284=21, srv670752142=77, srv1241730410=15, srv790158314=85, srv1273148539=17, srv2043427907=50, srv691803755=80, srv617308682=74, srv1971337060=48, srv1953285135=45, srv1006958827=0, srv904877743=92, srv1094681221=4, srv203102642=49, srv1467222976=26, srv604466476=73, srv389978366=60, srv1286374882=18, srv2110423993=54, srv1259325391=16, srv1094078004=3, srv478672610=68, srv800356455=86, srv40819841=62, srv828045274=88, srv1122970211=5, srv1229327611=14, srv928570216=95, srv68125779=78, srv440864950=66, srv902669367=91, srv431160466=65, srv1483704742=27, srv531140638=70, srv914552302=94, srv906312380=93, srv1837853444=40, srv743582204=82, srv1966798838=47, srv1456905324=25, srv1204458117=11, srv1911169586=43, srv980570543=99, srv1125648634=6, srv1583796260=32, srv1453892715=24, srv1391786565=22, srv781615027=84, srv548454019=72, srv970588901=97, srv1131673350=7, srv2053071672=51, srv1780760124=38, srv1598032065=33, srv97194566=98, srv1342325034=20, srv285283751=57, srv48227143=69, srv873794342=89, srv1716613309=35, srv226169559=56, srv18038785=39, srv1566826040=31, srv1954578515=46, srv1160797971=9, srv1338881444=19, srv2085386205=53, srv1192695892=10, srv404318557=61, srv2064207842=52, srv375695875=59, srv940066849=96, srv1145038612=8, srv1437409308=23, srv1772520018=37, srv46456461=67, srv343780200=58, srv650145184=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 20:55:19,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 20:55:19,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 20:55:19,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 20:55:19,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 20:55:19,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 20:55:19,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 20:55:19,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 20:55:19,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 20:55:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 20:55:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 20:55:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 20:55:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 20:55:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 20:55:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 20:55:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 20:55:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 20:55:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 20:55:19,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 20:55:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 20:55:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 20:55:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 20:55:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 20:55:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 20:55:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 20:55:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 20:55:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 20:55:19,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 20:55:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 20:55:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 20:55:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 20:55:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-11 20:55:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-11 20:55:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-11 20:55:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-11 20:55:19,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-11 20:55:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-11 20:55:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-11 20:55:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-11 20:55:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-11 20:55:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-11 20:55:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-11 20:55:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-11 20:55:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-11 20:55:19,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-11 20:55:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-11 20:55:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-11 20:55:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-11 20:55:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-11 20:55:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-11 20:55:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-11 20:55:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-11 20:55:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-11 20:55:19,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-11 20:55:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-11 20:55:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-11 20:55:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-11 20:55:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-11 20:55:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-11 20:55:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-11 20:55:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-11 20:55:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-11 20:55:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-11 20:55:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-11 20:55:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-11 20:55:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-11 20:55:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-11 20:55:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-11 20:55:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-11 20:55:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-11 20:55:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-11 20:55:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-11 20:55:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-11 20:55:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-11 20:55:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-11 20:55:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-11 20:55:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-11 20:55:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-11 20:55:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-11 20:55:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-11 20:55:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-11 20:55:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-11 20:55:19,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-11 20:55:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-11 20:55:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-11 20:55:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-11 20:55:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-11 20:55:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-11 20:55:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-11 20:55:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-11 20:55:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-11 20:55:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-11 20:55:19,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-11 20:55:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-11 20:55:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-11 20:55:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-11 20:55:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-11 20:55:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-11 20:55:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-11 20:55:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-11 20:55:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-11 20:55:19,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-11 20:55:19,757 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005102759975884884); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008206099398622182); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005102759975884884); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:55:19,758 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-11 20:55:19,758 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.592303202860134E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005102759975884884); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008206099398622182); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005102759975884884); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-11 20:55:20,192 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 541 ms to try 5000 different iterations. Found a solution that moves 246 regions; Going from a computed imbalance of 7.592303202860134E-4 to a new imbalance of 3.512416926536989E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.027333333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:55:20,194 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv906312380:82 , srv1716613309:83 , srv2114679593:83 , srv691803755:83 , srv743582204:83 , srv1006958827:84 , srv1072940955:84 , srv1093856147:84 , srv1094078004:84 , srv1125648634:84 , srv1145038612:84 , srv1192695892:84 , srv1204458117:84 , srv1212661309:84 , srv1218202541:84 , srv1273148539:84 , srv1286374882:84 , srv1342325034:84 , srv1354731284:84 , srv1467222976:84 , srv1483704742:84 , srv1501336588:84 , srv1531753662:84 , srv1555820673:84 , srv1566826040:84 , srv1583796260:84 , srv1780760124:84 , srv18038785:84 , srv1874430857:84 , srv1943725808:84 , srv1966798838:84 , srv203102642:84 , srv2053071672:84 , srv343780200:84 , srv375695875:84 , srv389978366:84 , srv404318557:84 , srv40819841:84 , srv440864950:84 , srv46456461:84 , srv478672610:84 , srv48227143:84 , srv531140638:84 , srv68125779:84 , srv691672014:84 , srv800356455:84 , srv810700249:84 , srv828045274:84 , srv873794342:84 , srv902669367:84 , srv914552302:84 , srv928570216:84 , srv940066849:84 , srv970588901:84 , srv980570543:84 , srv1094681221:85 , srv1122970211:85 , srv1131673350:85 , srv1160797971:85 , srv1229327611:85 , srv1259325391:85 , srv1338881444:85 , srv1391786565:85 , srv1453892715:85 , srv1456905324:85 , srv1598032065:85 , srv1642454058:85 , srv1772520018:85 , srv1837853444:85 , srv1903885467:85 , srv1911169586:85 , srv1953285135:85 , srv1954578515:85 , srv1971337060:85 , srv2043427907:85 , srv2085386205:85 , srv226169559:85 , srv285283751:85 , srv4259515:85 , srv42889655:85 , srv431160466:85 , srv539361443:85 , srv548454019:85 , srv604466476:85 , srv617308682:85 , srv625194687:85 , srv650145184:85 , srv670752142:85 , srv753533405:85 , srv781615027:85 , srv790158314:85 , srv97194566:85 , srv1241730410:86 , srv1437409308:86 , srv1743657961:86 , srv2064207842:86 , srv2110423993:86 , srv740730680:86 , srv904877743:86 , srv883329382:639 } 2023-11-11 20:55:20,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 20:55:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv753533405=83, srv1874430857=41, srv1531753662=29, srv1743657961=36, srv1943725808=44, srv1555820673=30, srv42889655=64, srv1072940955=1, srv625194687=75, srv2114679593=55, srv1642454058=34, srv539361443=71, srv740730680=81, srv1218202541=13, srv1501336588=28, srv1212661309=12, srv810700249=87, srv1093856147=2, srv883329382=90, srv4259515=63, srv691672014=79, srv1903885467=42, srv1354731284=21, srv670752142=77, srv1241730410=15, srv790158314=85, srv1273148539=17, srv2043427907=50, srv691803755=80, srv617308682=74, srv1971337060=48, srv1953285135=45, srv1006958827=0, srv904877743=92, srv1094681221=4, srv203102642=49, srv1467222976=26, srv604466476=73, srv389978366=60, srv1286374882=18, srv2110423993=54, srv1259325391=16, srv1094078004=3, srv478672610=68, srv800356455=86, srv40819841=62, srv828045274=88, srv1122970211=5, srv1229327611=14, srv928570216=95, srv68125779=78, srv440864950=66, srv902669367=91, srv431160466=65, srv1483704742=27, srv531140638=70, srv914552302=94, srv906312380=93, srv1837853444=40, srv743582204=82, srv1966798838=47, srv1456905324=25, srv1204458117=11, srv1911169586=43, srv980570543=99, srv1125648634=6, srv1583796260=32, srv1453892715=24, srv1391786565=22, srv781615027=84, srv548454019=72, srv970588901=97, srv1131673350=7, srv2053071672=51, srv1780760124=38, srv1598032065=33, srv97194566=98, srv1342325034=20, srv285283751=57, srv48227143=69, srv873794342=89, srv1716613309=35, srv226169559=56, srv18038785=39, srv1566826040=31, srv1954578515=46, srv1160797971=9, srv1338881444=19, srv2085386205=53, srv1192695892=10, srv404318557=61, srv2064207842=52, srv375695875=59, srv940066849=96, srv1145038612=8, srv1437409308=23, srv1772520018=37, srv46456461=67, srv343780200=58, srv650145184=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 20:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 20:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 20:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 20:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 20:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 20:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 20:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 20:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 20:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 20:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 20:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 20:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 20:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 20:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 20:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 20:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 20:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 20:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 20:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 20:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 20:55:20,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 20:55:20,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 20:55:20,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 20:55:20,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 20:55:20,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 20:55:20,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 20:55:20,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 20:55:20,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 20:55:20,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 20:55:20,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 20:55:20,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-11 20:55:20,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-11 20:55:20,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-11 20:55:20,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-11 20:55:20,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-11 20:55:20,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-11 20:55:20,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-11 20:55:20,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-11 20:55:20,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-11 20:55:20,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-11 20:55:20,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-11 20:55:20,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-11 20:55:20,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-11 20:55:20,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-11 20:55:20,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-11 20:55:20,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-11 20:55:20,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-11 20:55:20,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-11 20:55:20,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-11 20:55:20,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-11 20:55:20,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-11 20:55:20,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-11 20:55:20,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-11 20:55:20,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-11 20:55:20,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-11 20:55:20,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-11 20:55:20,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-11 20:55:20,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-11 20:55:20,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-11 20:55:20,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-11 20:55:20,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-11 20:55:20,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-11 20:55:20,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-11 20:55:20,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-11 20:55:20,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-11 20:55:20,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-11 20:55:20,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-11 20:55:20,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-11 20:55:20,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-11 20:55:20,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-11 20:55:20,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-11 20:55:20,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-11 20:55:20,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-11 20:55:20,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-11 20:55:20,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-11 20:55:20,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-11 20:55:20,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-11 20:55:20,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-11 20:55:20,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-11 20:55:20,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-11 20:55:20,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-11 20:55:20,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-11 20:55:20,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-11 20:55:20,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-11 20:55:20,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-11 20:55:20,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-11 20:55:20,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-11 20:55:20,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-11 20:55:20,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-11 20:55:20,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-11 20:55:20,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-11 20:55:20,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-11 20:55:20,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-11 20:55:20,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-11 20:55:20,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-11 20:55:20,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-11 20:55:20,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-11 20:55:20,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-11 20:55:20,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-11 20:55:20,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-11 20:55:20,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-11 20:55:20,333 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:55:20,333 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-11-11 20:55:20,333 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.340164044902328E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-11-11 20:55:20,597 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 359 ms to try 5000 different iterations. Found a solution that moves 27 regions; Going from a computed imbalance of 3.340164044902328E-5 to a new imbalance of 7.57681554113109E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.003); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:55:20,597 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1131673350:83 , srv1241730410:83 , srv1437409308:83 , srv1566826040:83 , srv1743657961:83 , srv2064207842:83 , srv4259515:83 , srv431160466:83 , srv740730680:83 , srv753533405:83 , srv781615027:83 , srv904877743:83 , srv1006958827:84 , srv1072940955:84 , srv1094078004:84 , srv1094681221:84 , srv1122970211:84 , srv1125648634:84 , srv1145038612:84 , srv1160797971:84 , srv1192695892:84 , srv1204458117:84 , srv1212661309:84 , srv1218202541:84 , srv1229327611:84 , srv1259325391:84 , srv1273148539:84 , srv1338881444:84 , srv1342325034:84 , srv1354731284:84 , srv1391786565:84 , srv1453892715:84 , srv1456905324:84 , srv1467222976:84 , srv1483704742:84 , srv1501336588:84 , srv1531753662:84 , srv1555820673:84 , srv1583796260:84 , srv1598032065:84 , srv1642454058:84 , srv1716613309:84 , srv1772520018:84 , srv1837853444:84 , srv1874430857:84 , srv1903885467:84 , srv1911169586:84 , srv1943725808:84 , srv1953285135:84 , srv1954578515:84 , srv1966798838:84 , srv1971337060:84 , srv203102642:84 , srv2053071672:84 , srv2085386205:84 , srv2110423993:84 , srv2114679593:84 , srv226169559:84 , srv285283751:84 , srv375695875:84 , srv389978366:84 , srv404318557:84 , srv42889655:84 , srv46456461:84 , srv478672610:84 , srv48227143:84 , srv531140638:84 , srv539361443:84 , srv548454019:84 , srv604466476:84 , srv617308682:84 , srv625194687:84 , srv650145184:84 , srv670752142:84 , srv68125779:84 , srv691672014:84 , srv691803755:84 , srv790158314:84 , srv810700249:84 , srv828045274:84 , srv906312380:84 , srv914552302:84 , srv928570216:84 , srv940066849:84 , srv970588901:84 , srv97194566:84 , srv980570543:84 , srv1093856147:85 , srv1286374882:85 , srv1780760124:85 , srv18038785:85 , srv2043427907:85 , srv343780200:85 , srv40819841:85 , srv440864950:85 , srv743582204:85 , srv800356455:85 , srv873794342:85 , srv902669367:85 , srv883329382:684 } 2023-11-11 20:55:20,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 20:55:20,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv753533405=83, srv1874430857=41, srv1531753662=29, srv1743657961=36, srv1943725808=44, srv1555820673=30, srv42889655=64, srv1072940955=1, srv625194687=75, srv2114679593=55, srv1642454058=34, srv539361443=71, srv740730680=81, srv1218202541=13, srv1501336588=28, srv1212661309=12, srv810700249=87, srv1093856147=2, srv883329382=90, srv4259515=63, srv691672014=79, srv1903885467=42, srv1354731284=21, srv670752142=77, srv1241730410=15, srv790158314=85, srv1273148539=17, srv2043427907=50, srv691803755=80, srv617308682=74, srv1971337060=48, srv1953285135=45, srv1006958827=0, srv904877743=92, srv1094681221=4, srv203102642=49, srv1467222976=26, srv604466476=73, srv389978366=60, srv1286374882=18, srv2110423993=54, srv1259325391=16, srv1094078004=3, srv478672610=68, srv800356455=86, srv40819841=62, srv828045274=88, srv1122970211=5, srv1229327611=14, srv928570216=95, srv68125779=78, srv440864950=66, srv902669367=91, srv431160466=65, srv1483704742=27, srv531140638=70, srv914552302=94, srv906312380=93, srv1837853444=40, srv743582204=82, srv1966798838=47, srv1456905324=25, srv1204458117=11, srv1911169586=43, srv980570543=99, srv1125648634=6, srv1583796260=32, srv1453892715=24, srv1391786565=22, srv781615027=84, srv548454019=72, srv970588901=97, srv1131673350=7, srv2053071672=51, srv1780760124=38, srv1598032065=33, srv97194566=98, srv1342325034=20, srv285283751=57, srv48227143=69, srv873794342=89, srv1716613309=35, srv226169559=56, srv18038785=39, srv1566826040=31, srv1954578515=46, srv1160797971=9, srv1338881444=19, srv2085386205=53, srv1192695892=10, srv404318557=61, srv2064207842=52, srv375695875=59, srv940066849=96, srv1145038612=8, srv1437409308=23, srv1772520018=37, srv46456461=67, srv343780200=58, srv650145184=76} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 20:55:20,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 20:55:20,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 20:55:20,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 20:55:20,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-11-11 20:55:20,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-11-11 20:55:20,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-11-11 20:55:20,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-11-11 20:55:20,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-11-11 20:55:20,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-11-11 20:55:20,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-11-11 20:55:20,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-11-11 20:55:20,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-11-11 20:55:20,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-11-11 20:55:20,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-11-11 20:55:20,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-11-11 20:55:20,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-11-11 20:55:20,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-11-11 20:55:20,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-11-11 20:55:20,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-11-11 20:55:20,716 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:55:20,717 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1131673350:83 , srv1241730410:83 , srv1437409308:83 , srv1566826040:83 , srv1743657961:83 , srv2064207842:83 , srv4259515:83 , srv431160466:83 , srv740730680:83 , srv753533405:83 , srv781615027:83 , srv904877743:83 , srv1006958827:84 , srv1072940955:84 , srv1094078004:84 , srv1094681221:84 , srv1122970211:84 , srv1125648634:84 , srv1145038612:84 , srv1160797971:84 , srv1192695892:84 , srv1204458117:84 , srv1212661309:84 , srv1218202541:84 , srv1229327611:84 , srv1259325391:84 , srv1273148539:84 , srv1338881444:84 , srv1342325034:84 , srv1354731284:84 , srv1391786565:84 , srv1453892715:84 , srv1456905324:84 , srv1467222976:84 , srv1483704742:84 , srv1501336588:84 , srv1531753662:84 , srv1555820673:84 , srv1583796260:84 , srv1598032065:84 , srv1642454058:84 , srv1716613309:84 , srv1772520018:84 , srv1837853444:84 , srv1874430857:84 , srv1903885467:84 , srv1911169586:84 , srv1943725808:84 , srv1953285135:84 , srv1954578515:84 , srv1966798838:84 , srv1971337060:84 , srv203102642:84 , srv2053071672:84 , srv2085386205:84 , srv2110423993:84 , srv2114679593:84 , srv226169559:84 , srv285283751:84 , srv375695875:84 , srv389978366:84 , srv404318557:84 , srv42889655:84 , srv46456461:84 , srv478672610:84 , srv48227143:84 , srv531140638:84 , srv539361443:84 , srv548454019:84 , srv604466476:84 , srv617308682:84 , srv625194687:84 , srv650145184:84 , srv670752142:84 , srv68125779:84 , srv691672014:84 , srv691803755:84 , srv790158314:84 , srv810700249:84 , srv828045274:84 , srv906312380:84 , srv914552302:84 , srv928570216:84 , srv940066849:84 , srv970588901:84 , srv97194566:84 , srv980570543:84 , srv1093856147:85 , srv1286374882:85 , srv1780760124:85 , srv18038785:85 , srv2043427907:85 , srv343780200:85 , srv40819841:85 , srv440864950:85 , srv743582204:85 , srv800356455:85 , srv873794342:85 , srv902669367:85 , srv883329382:684 } 2023-11-11 20:55:20,739 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=122 (was 97) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=1138 (was 1636) 2023-11-11 20:55:20,746 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=122, ProcessCount=170, AvailableMemoryMB=1138 2023-11-11 20:55:20,747 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-11-11 20:55:20,747 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-11-11 20:55:20,756 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1069670707:84 , srv1100863126:84 , srv1112604187:84 , srv1248279299:84 , srv1316653075:84 , srv1383901416:84 , srv1524186459:84 , srv1556888786:84 , srv1569453041:84 , srv1754118919:84 , srv1811148587:84 , srv1817569175:84 , srv1882078212:84 , srv1923009101:84 , srv1963680019:84 , srv1995343703:84 , srv2026840478:84 , srv20467090:84 , srv2052562488:84 , srv220035720:84 , srv251640164:84 , srv288996878:84 , srv399129204:84 , srv407138498:84 , srv691159076:84 , srv793781885:84 , srv831981275:84 , srv951373646:84 , srv993396240:84 , srv612947766:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-11-11 20:55:20,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 20:55:20,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1524186459=6, srv1383901416=5, srv612947766=24, srv793781885=26, srv1248279299=3, srv2026840478=16, srv831981275=27, srv251640164=20, srv1112604187=2, srv2052562488=18, srv1923009101=13, srv691159076=25, srv1882078212=12, srv951373646=28, srv20467090=17, srv993396240=29, srv1316653075=4, srv1963680019=14, srv288996878=21, srv1069670707=0, srv1569453041=8, srv1817569175=11, srv220035720=19, srv1556888786=7, srv1754118919=9, srv1811148587=10, srv1995343703=15, srv1100863126=1, srv399129204=22, srv407138498=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 20:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 20:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 20:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 20:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 20:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 20:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 20:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 20:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 20:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 20:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 20:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 20:55:20,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-11-11 20:55:20,781 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0896597808049047, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:55:20,781 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912475558305935, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0896597808049047, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-11-11 20:57:20,769 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5314707 different iterations. Found a solution that moves 1864 regions; Going from a computed imbalance of 0.9912475558305935 to a new imbalance of 4.3506689887128686E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6903703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:57:20,776 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1069670707:90 , srv1100863126:90 , srv1112604187:90 , srv1248279299:90 , srv1316653075:90 , srv1383901416:90 , srv1524186459:90 , srv1556888786:90 , srv1569453041:90 , srv1754118919:90 , srv1811148587:90 , srv1817569175:90 , srv1882078212:90 , srv1923009101:90 , srv1963680019:90 , srv1995343703:90 , srv2026840478:90 , srv20467090:90 , srv2052562488:90 , srv220035720:90 , srv251640164:90 , srv288996878:90 , srv399129204:90 , srv407138498:90 , srv612947766:90 , srv691159076:90 , srv793781885:90 , srv831981275:90 , srv951373646:90 , srv993396240:90 } 2023-11-11 20:57:20,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-11-11 20:57:20,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1524186459=6, srv1383901416=5, srv612947766=24, srv793781885=26, srv1248279299=3, srv2026840478=16, srv831981275=27, srv251640164=20, srv1112604187=2, srv2052562488=18, srv1923009101=13, srv691159076=25, srv1882078212=12, srv951373646=28, srv20467090=17, srv993396240=29, srv1316653075=4, srv1963680019=14, srv288996878=21, srv1069670707=0, srv1569453041=8, srv1817569175=11, srv220035720=19, srv1556888786=7, srv1754118919=9, srv1811148587=10, srv1995343703=15, srv1100863126=1, srv399129204=22, srv407138498=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-11-11 20:57:20,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-11-11 20:57:20,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-11-11 20:57:20,794 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-11-11 20:57:20,794 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1069670707:90 , srv1100863126:90 , srv1112604187:90 , srv1248279299:90 , srv1316653075:90 , srv1383901416:90 , srv1524186459:90 , srv1556888786:90 , srv1569453041:90 , srv1754118919:90 , srv1811148587:90 , srv1817569175:90 , srv1882078212:90 , srv1923009101:90 , srv1963680019:90 , srv1995343703:90 , srv2026840478:90 , srv20467090:90 , srv2052562488:90 , srv220035720:90 , srv251640164:90 , srv288996878:90 , srv399129204:90 , srv407138498:90 , srv612947766:90 , srv691159076:90 , srv793781885:90 , srv831981275:90 , srv951373646:90 , srv993396240:90 } 2023-11-11 20:57:20,806 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=252 (was 122) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 170), AvailableMemoryMB=1323 (was 1138) - AvailableMemoryMB LEAK? -