2023-04-28 18:55:44,892 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-04-28 18:55:45,271 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 18:55:45,398 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-28 18:55:45,475 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-28 18:55:45,485 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-28 18:55:45,503 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=401, ProcessCount=176, AvailableMemoryMB=12547 2023-04-28 18:55:45,506 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-28 18:55:45,506 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-28 18:55:45,677 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001874292:84 , srv102600028:84 , srv1031569012:84 , srv1035115816:84 , srv1059143689:84 , srv1080975212:84 , srv1094003071:84 , srv1108549330:84 , srv1122887864:84 , srv1129187718:84 , srv1167752309:84 , srv1176044827:84 , srv1178467675:84 , srv1202839180:84 , srv1205079912:84 , srv1211983039:84 , srv1220013637:84 , srv1234839184:84 , srv1342331011:84 , srv1360493600:84 , srv1370731162:84 , srv1376730872:84 , srv1391941215:84 , srv1397304305:84 , srv1473702985:84 , srv1496597095:84 , srv1557894940:84 , srv1570256225:84 , srv1593942592:84 , srv1612378940:84 , srv1614137333:84 , srv1634907019:84 , srv1636683345:84 , srv1664138818:84 , srv1667533585:84 , srv1702902910:84 , srv1709880603:84 , srv1710946337:84 , srv1740590547:84 , srv1819221209:84 , srv1849811035:84 , srv1867284163:84 , srv1870386760:84 , srv1878589687:84 , srv1879399424:84 , srv1892285893:84 , srv1929274594:84 , srv19890972:84 , srv2025185998:84 , srv2033130284:84 , srv2072817726:84 , srv2074320680:84 , srv208385227:84 , srv2084930318:84 , srv2085917343:84 , srv2090839365:84 , srv212027338:84 , srv2141584619:84 , srv2145214720:84 , srv234951261:84 , srv25100633:84 , srv261689959:84 , srv27723583:84 , srv282062910:84 , srv287168230:84 , srv296797387:84 , srv332174597:84 , srv341325817:84 , srv396545723:84 , srv397250822:84 , srv425250220:84 , srv429443379:84 , srv44548987:84 , srv461111322:84 , srv475379161:84 , srv476867875:84 , srv510101192:84 , srv529002561:84 , srv53516150:84 , srv589944039:84 , srv600568230:84 , srv605116776:84 , srv638296674:84 , srv639392223:84 , srv642203643:84 , srv672401494:84 , srv678097564:84 , srv699291259:84 , srv709688627:84 , srv715627078:84 , srv799381491:84 , srv826291460:84 , srv847123025:84 , srv853148212:84 , srv874605768:84 , srv877565223:84 , srv896657461:84 , srv922413485:84 , srv98165554:84 , srv1236328696:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-04-28 18:55:45,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-28 18:55:45,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv672401494=86, srv208385227=53, srv212027338=57, srv341325817=68, srv877565223=96, srv1202839180=13, srv1001874292=0, srv287168230=65, srv1710946337=38, srv1167752309=10, srv2025185998=49, srv475379161=75, srv102600028=1, srv678097564=87, srv1220013637=16, srv1473702985=25, srv896657461=97, srv1867284163=42, srv847123025=93, srv1612378940=30, srv1702902910=36, srv922413485=98, srv1664138818=34, srv1570256225=28, srv2084930318=54, srv282062910=64, srv1035115816=3, srv1819221209=40, srv19890972=48, srv699291259=88, srv799381491=91, srv1234839184=17, srv1391941215=23, srv332174597=67, srv1740590547=39, srv1370731162=21, srv1376730872=22, srv1709880603=37, srv2090839365=56, srv1031569012=2, srv2074320680=52, srv1342331011=19, srv25100633=61, srv715627078=90, srv1360493600=20, srv98165554=99, srv1879399424=45, srv296797387=66, srv510101192=77, srv27723583=63, srv1129187718=9, srv1176044827=11, srv638296674=83, srv1849811035=41, srv2145214720=59, srv639392223=84, srv874605768=95, srv1496597095=26, srv2085917343=55, srv429443379=72, srv44548987=73, srv589944039=80, srv234951261=60, srv1878589687=44, srv1636683345=33, srv1870386760=43, srv2033130284=50, srv2141584619=58, srv396545723=69, srv476867875=76, srv600568230=81, srv1557894940=27, srv826291460=92, srv461111322=74, srv397250822=70, srv1122887864=8, srv642203643=85, srv1634907019=32, srv1929274594=47, srv1205079912=14, srv1614137333=31, srv605116776=82, srv261689959=62, srv1178467675=12, srv1397304305=24, srv1236328696=18, srv709688627=89, srv2072817726=51, srv529002561=78, srv1211983039=15, srv1108549330=7, srv53516150=79, srv1667533585=35, srv1059143689=4, srv1593942592=29, srv1094003071=6, srv1892285893=46, srv853148212=94, srv1080975212=5, srv425250220=71} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-28 18:55:45,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-28 18:55:45,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-28 18:55:45,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-28 18:55:45,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-28 18:55:45,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-28 18:55:45,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-28 18:55:45,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-28 18:55:45,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-28 18:55:45,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-28 18:55:45,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-28 18:55:45,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-28 18:55:45,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-28 18:55:45,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-28 18:55:45,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-28 18:55:45,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-28 18:55:45,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-28 18:55:45,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-28 18:55:45,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-28 18:55:45,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-28 18:55:45,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-28 18:55:45,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-28 18:55:45,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-28 18:55:45,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-28 18:55:45,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-28 18:55:45,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-28 18:55:45,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-28 18:55:45,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-28 18:55:45,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-28 18:55:45,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-28 18:55:45,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-28 18:55:45,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-28 18:55:45,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-28 18:55:45,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-28 18:55:45,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-28 18:55:45,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-28 18:55:45,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-28 18:55:45,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-28 18:55:45,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-28 18:55:45,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-28 18:55:45,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-28 18:55:45,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-28 18:55:45,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-28 18:55:45,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-28 18:55:45,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-28 18:55:45,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-28 18:55:45,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-28 18:55:45,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-28 18:55:45,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-28 18:55:45,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-28 18:55:45,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-28 18:55:45,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-28 18:55:45,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-28 18:55:45,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-28 18:55:45,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-28 18:55:45,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-28 18:55:45,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-28 18:55:45,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-28 18:55:45,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-28 18:55:45,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-28 18:55:45,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-28 18:55:45,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-28 18:55:45,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-28 18:55:45,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-28 18:55:45,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-28 18:55:45,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-28 18:55:45,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-28 18:55:45,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-28 18:55:45,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-28 18:55:45,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-28 18:55:45,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-28 18:55:45,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-28 18:55:45,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-28 18:55:45,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-28 18:55:45,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-28 18:55:45,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-28 18:55:45,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-28 18:55:45,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-28 18:55:45,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-28 18:55:45,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-28 18:55:45,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-28 18:55:45,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-28 18:55:45,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-28 18:55:45,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-28 18:55:45,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-28 18:55:45,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-28 18:55:45,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-28 18:55:45,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-28 18:55:45,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-28 18:55:45,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-28 18:55:45,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-28 18:55:45,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-28 18:55:45,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-28 18:55:45,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-28 18:55:45,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-28 18:55:45,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-28 18:55:45,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-28 18:55:45,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-28 18:55:45,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-28 18:55:45,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-28 18:55:45,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-28 18:55:45,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-28 18:55:46,039 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:55:46,039 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-28 18:55:46,039 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-28 18:55:46,424 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 553 ms to try 5000 different iterations. Found a solution that moves 4685 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18029568229297183. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0409270304709488, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05732730373117524, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5205555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0409270304709488, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1755, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24225, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:55:46,446 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv287168230:82 , srv1108549330:83 , srv212027338:83 , srv2145214720:83 , srv396545723:83 , srv510101192:83 , srv102600028:84 , srv1094003071:84 , srv1176044827:84 , srv1202839180:84 , srv1370731162:84 , srv1473702985:84 , srv1740590547:84 , srv1892285893:84 , srv2141584619:84 , srv332174597:84 , srv425250220:84 , srv44548987:84 , srv53516150:84 , srv600568230:84 , srv605116776:84 , srv638296674:84 , srv874605768:84 , srv1001874292:85 , srv1080975212:85 , srv1167752309:85 , srv1178467675:85 , srv1205079912:85 , srv1664138818:85 , srv1702902910:85 , srv1819221209:85 , srv1849811035:85 , srv1929274594:85 , srv475379161:85 , srv529002561:85 , srv589944039:85 , srv672401494:85 , srv922413485:85 , srv1031569012:86 , srv1360493600:86 , srv1397304305:86 , srv1614137333:86 , srv1634907019:86 , srv1667533585:86 , srv1709880603:86 , srv1870386760:86 , srv1879399424:86 , srv2025185998:86 , srv208385227:86 , srv2084930318:86 , srv2085917343:86 , srv234951261:86 , srv341325817:86 , srv476867875:86 , srv709688627:86 , srv715627078:86 , srv826291460:86 , srv877565223:86 , srv896657461:86 , srv1035115816:87 , srv1059143689:87 , srv1220013637:87 , srv1342331011:87 , srv1391941215:87 , srv1557894940:87 , srv1570256225:87 , srv1878589687:87 , srv2072817726:87 , srv261689959:87 , srv296797387:87 , srv397250822:87 , srv642203643:87 , srv699291259:87 , srv1122887864:88 , srv1376730872:88 , srv1867284163:88 , srv19890972:88 , srv25100633:88 , srv27723583:88 , srv461111322:88 , srv853148212:88 , srv98165554:88 , srv1211983039:89 , srv1234839184:89 , srv1496597095:89 , srv1710946337:89 , srv282062910:89 , srv678097564:89 , srv799381491:89 , srv847123025:89 , srv1129187718:90 , srv1636683345:90 , srv2074320680:90 , srv429443379:90 , srv639392223:90 , srv1593942592:91 , srv1612378940:91 , srv2090839365:92 , srv2033130284:94 , srv1236328696:454 } 2023-04-28 18:55:46,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-28 18:55:46,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv672401494=86, srv208385227=53, srv212027338=57, srv341325817=68, srv877565223=96, srv1202839180=13, srv1001874292=0, srv287168230=65, srv1710946337=38, srv1167752309=10, srv2025185998=49, srv475379161=75, srv102600028=1, srv678097564=87, srv1220013637=16, srv1473702985=25, srv896657461=97, srv1867284163=42, srv847123025=93, srv1612378940=30, srv1702902910=36, srv922413485=98, srv1664138818=34, srv1570256225=28, srv2084930318=54, srv282062910=64, srv1035115816=3, srv1819221209=40, srv19890972=48, srv699291259=88, srv799381491=91, srv1234839184=17, srv1391941215=23, srv332174597=67, srv1740590547=39, srv1370731162=21, srv1376730872=22, srv1709880603=37, srv2090839365=56, srv1031569012=2, srv2074320680=52, srv1342331011=19, srv25100633=61, srv715627078=90, srv1360493600=20, srv98165554=99, srv1879399424=45, srv296797387=66, srv510101192=77, srv27723583=63, srv1129187718=9, srv1176044827=11, srv638296674=83, srv1849811035=41, srv2145214720=59, srv639392223=84, srv874605768=95, srv1496597095=26, srv2085917343=55, srv429443379=72, srv44548987=73, srv589944039=80, srv234951261=60, srv1878589687=44, srv1636683345=33, srv1870386760=43, srv2033130284=50, srv2141584619=58, srv396545723=69, srv476867875=76, srv600568230=81, srv1557894940=27, srv826291460=92, srv461111322=74, srv397250822=70, srv1122887864=8, srv642203643=85, srv1634907019=32, srv1929274594=47, srv1205079912=14, srv1614137333=31, srv605116776=82, srv261689959=62, srv1178467675=12, srv1397304305=24, srv1236328696=18, srv709688627=89, srv2072817726=51, srv529002561=78, srv1211983039=15, srv1108549330=7, srv53516150=79, srv1667533585=35, srv1059143689=4, srv1593942592=29, srv1094003071=6, srv1892285893=46, srv853148212=94, srv1080975212=5, srv425250220=71} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-28 18:55:46,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-28 18:55:46,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-28 18:55:46,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-28 18:55:46,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-28 18:55:46,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-28 18:55:46,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-28 18:55:46,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-28 18:55:46,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-28 18:55:46,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-28 18:55:46,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-28 18:55:46,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-28 18:55:46,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-28 18:55:46,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-28 18:55:46,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-28 18:55:46,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-28 18:55:46,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-28 18:55:46,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-28 18:55:46,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-28 18:55:46,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-28 18:55:46,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-28 18:55:46,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-28 18:55:46,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-28 18:55:46,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-28 18:55:46,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-28 18:55:46,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-28 18:55:46,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-28 18:55:46,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-28 18:55:46,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-28 18:55:46,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-28 18:55:46,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-28 18:55:46,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-28 18:55:46,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-28 18:55:46,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-28 18:55:46,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-28 18:55:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-28 18:55:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-28 18:55:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-28 18:55:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-28 18:55:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-28 18:55:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-28 18:55:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-28 18:55:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-28 18:55:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-28 18:55:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-28 18:55:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-28 18:55:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-28 18:55:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-28 18:55:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-28 18:55:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-28 18:55:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-28 18:55:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-28 18:55:46,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-28 18:55:46,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-28 18:55:46,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-28 18:55:46,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-28 18:55:46,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-28 18:55:46,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-28 18:55:46,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-28 18:55:46,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-28 18:55:46,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-28 18:55:46,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-28 18:55:46,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-28 18:55:46,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-28 18:55:46,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-28 18:55:46,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-28 18:55:46,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-28 18:55:46,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-28 18:55:46,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-28 18:55:46,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-28 18:55:46,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-28 18:55:46,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-28 18:55:46,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-28 18:55:46,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-28 18:55:46,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-28 18:55:46,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-28 18:55:46,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-28 18:55:46,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-28 18:55:46,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-28 18:55:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-28 18:55:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-28 18:55:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-28 18:55:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-28 18:55:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-28 18:55:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-28 18:55:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-28 18:55:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-28 18:55:46,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-28 18:55:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-28 18:55:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-28 18:55:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-28 18:55:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-28 18:55:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-28 18:55:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-28 18:55:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-28 18:55:46,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-28 18:55:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-28 18:55:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-28 18:55:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-28 18:55:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-28 18:55:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-28 18:55:46,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-28 18:55:46,670 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0409270304709488, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05732730373117524, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0409270304709488, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1755, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24225, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:55:46,671 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-28 18:55:46,671 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.180262877221815, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0409270304709488, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05732730373117524, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0409270304709488, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1755, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24225, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-28 18:55:47,070 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 551 ms to try 5000 different iterations. Found a solution that moves 2659 regions; Going from a computed imbalance of 0.180262877221815 to a new imbalance of 0.015467965397133817. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02006971264757741); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035659570079736025, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29544444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02006971264757741); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:55:47,088 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2090839365:78 , srv1636683345:80 , srv2033130284:80 , srv429443379:81 , srv1496597095:82 , srv1593942592:82 , srv1612378940:82 , srv1710946337:82 , srv2074320680:82 , srv25100633:82 , srv847123025:82 , srv1211983039:83 , srv1234839184:83 , srv896657461:83 , srv98165554:83 , srv1059143689:84 , srv1122887864:84 , srv1129187718:84 , srv1342331011:84 , srv1376730872:84 , srv1557894940:84 , srv1614137333:84 , srv1879399424:84 , srv2084930318:84 , srv27723583:84 , srv282062910:84 , srv296797387:84 , srv341325817:84 , srv475379161:84 , srv642203643:84 , srv678097564:84 , srv1220013637:85 , srv1360493600:85 , srv1391941215:85 , srv1397304305:85 , srv1664138818:85 , srv1929274594:85 , srv2025185998:85 , srv261689959:85 , srv461111322:85 , srv853148212:85 , srv1031569012:86 , srv1035115816:86 , srv1178467675:86 , srv1205079912:86 , srv1709880603:86 , srv1867284163:86 , srv1870386760:86 , srv1878589687:86 , srv208385227:86 , srv2085917343:86 , srv234951261:86 , srv672401494:86 , srv715627078:86 , srv799381491:86 , srv922413485:86 , srv1001874292:87 , srv102600028:87 , srv1094003071:87 , srv1167752309:87 , srv1473702985:87 , srv1667533585:87 , srv1702902910:87 , srv1740590547:87 , srv1819221209:87 , srv1849811035:87 , srv1892285893:87 , srv2072817726:87 , srv2141584619:87 , srv397250822:87 , srv510101192:87 , srv53516150:87 , srv605116776:87 , srv699291259:87 , srv1080975212:88 , srv1176044827:88 , srv1370731162:88 , srv1570256225:88 , srv19890972:88 , srv476867875:88 , srv529002561:88 , srv589944039:88 , srv638296674:88 , srv639392223:88 , srv709688627:88 , srv877565223:88 , srv1108549330:89 , srv1202839180:89 , srv287168230:89 , srv425250220:89 , srv826291460:89 , srv874605768:89 , srv1634907019:90 , srv212027338:90 , srv2145214720:90 , srv396545723:90 , srv332174597:91 , srv44548987:91 , srv600568230:95 , srv1236328696:498 } 2023-04-28 18:55:47,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-28 18:55:47,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv672401494=86, srv208385227=53, srv212027338=57, srv341325817=68, srv877565223=96, srv1202839180=13, srv1001874292=0, srv287168230=65, srv1710946337=38, srv1167752309=10, srv2025185998=49, srv475379161=75, srv102600028=1, srv678097564=87, srv1220013637=16, srv1473702985=25, srv896657461=97, srv1867284163=42, srv847123025=93, srv1612378940=30, srv1702902910=36, srv922413485=98, srv1664138818=34, srv1570256225=28, srv2084930318=54, srv282062910=64, srv1035115816=3, srv1819221209=40, srv19890972=48, srv699291259=88, srv799381491=91, srv1234839184=17, srv1391941215=23, srv332174597=67, srv1740590547=39, srv1370731162=21, srv1376730872=22, srv1709880603=37, srv2090839365=56, srv1031569012=2, srv2074320680=52, srv1342331011=19, srv25100633=61, srv715627078=90, srv1360493600=20, srv98165554=99, srv1879399424=45, srv296797387=66, srv510101192=77, srv27723583=63, srv1129187718=9, srv1176044827=11, srv638296674=83, srv1849811035=41, srv2145214720=59, srv639392223=84, srv874605768=95, srv1496597095=26, srv2085917343=55, srv429443379=72, srv44548987=73, srv589944039=80, srv234951261=60, srv1878589687=44, srv1636683345=33, srv1870386760=43, srv2033130284=50, srv2141584619=58, srv396545723=69, srv476867875=76, srv600568230=81, srv1557894940=27, srv826291460=92, srv461111322=74, srv397250822=70, srv1122887864=8, srv642203643=85, srv1634907019=32, srv1929274594=47, srv1205079912=14, srv1614137333=31, srv605116776=82, srv261689959=62, srv1178467675=12, srv1397304305=24, srv1236328696=18, srv709688627=89, srv2072817726=51, srv529002561=78, srv1211983039=15, srv1108549330=7, srv53516150=79, srv1667533585=35, srv1059143689=4, srv1593942592=29, srv1094003071=6, srv1892285893=46, srv853148212=94, srv1080975212=5, srv425250220=71} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-28 18:55:47,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-28 18:55:47,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-28 18:55:47,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-28 18:55:47,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-28 18:55:47,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-28 18:55:47,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-28 18:55:47,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-28 18:55:47,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-28 18:55:47,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-28 18:55:47,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-28 18:55:47,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-28 18:55:47,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-28 18:55:47,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-28 18:55:47,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-28 18:55:47,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-28 18:55:47,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-28 18:55:47,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-28 18:55:47,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-28 18:55:47,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-28 18:55:47,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-28 18:55:47,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-28 18:55:47,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-28 18:55:47,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-28 18:55:47,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-28 18:55:47,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-28 18:55:47,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-28 18:55:47,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-28 18:55:47,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-28 18:55:47,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-28 18:55:47,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-28 18:55:47,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-28 18:55:47,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-28 18:55:47,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-28 18:55:47,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-28 18:55:47,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-28 18:55:47,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-28 18:55:47,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-28 18:55:47,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-28 18:55:47,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-28 18:55:47,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-28 18:55:47,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-28 18:55:47,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-28 18:55:47,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-28 18:55:47,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-28 18:55:47,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-28 18:55:47,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-28 18:55:47,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-28 18:55:47,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-28 18:55:47,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-28 18:55:47,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-28 18:55:47,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-28 18:55:47,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-28 18:55:47,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-28 18:55:47,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-28 18:55:47,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-28 18:55:47,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-28 18:55:47,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-28 18:55:47,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-28 18:55:47,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-28 18:55:47,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-28 18:55:47,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-28 18:55:47,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-28 18:55:47,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-28 18:55:47,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-28 18:55:47,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-28 18:55:47,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-28 18:55:47,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-28 18:55:47,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-28 18:55:47,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-28 18:55:47,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-28 18:55:47,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-28 18:55:47,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-28 18:55:47,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-28 18:55:47,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-28 18:55:47,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-28 18:55:47,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-28 18:55:47,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-28 18:55:47,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-28 18:55:47,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-28 18:55:47,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-28 18:55:47,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-28 18:55:47,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-28 18:55:47,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-28 18:55:47,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-28 18:55:47,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-28 18:55:47,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-28 18:55:47,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-28 18:55:47,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-28 18:55:47,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-28 18:55:47,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-28 18:55:47,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-28 18:55:47,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-28 18:55:47,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-28 18:55:47,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-28 18:55:47,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-28 18:55:47,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-28 18:55:47,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-28 18:55:47,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-28 18:55:47,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-28 18:55:47,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-28 18:55:47,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-28 18:55:47,280 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02006971264757741); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035659570079736025, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02006971264757741); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:55:47,280 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-28 18:55:47,281 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015449346681188022, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02006971264757741); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035659570079736025, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02006971264757741); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-28 18:55:47,713 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 568 ms to try 5000 different iterations. Found a solution that moves 992 regions; Going from a computed imbalance of 0.015449346681188022 to a new imbalance of 0.001149616617693039. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006122578771996661); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0089643263564985); MoveCostFunction : (multiplier=7.0, imbalance=0.11022222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006122578771996661); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:55:47,723 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv600568230:80 , srv639392223:80 , srv1634907019:81 , srv19890972:81 , srv1570256225:82 , srv332174597:82 , srv44548987:82 , srv799381491:82 , srv826291460:82 , srv1129187718:83 , srv1867284163:83 , srv2033130284:83 , srv2072817726:83 , srv397250822:83 , srv476867875:83 , srv709688627:83 , srv1035115816:84 , srv1080975212:84 , srv1612378940:84 , srv1878589687:84 , srv208385227:84 , srv212027338:84 , srv2145214720:84 , srv396545723:84 , srv461111322:84 , srv529002561:84 , srv699291259:84 , srv1001874292:85 , srv102600028:85 , srv1108549330:85 , srv1122887864:85 , srv1167752309:85 , srv1176044827:85 , srv1202839180:85 , srv1220013637:85 , srv1473702985:85 , srv1593942592:85 , srv1710946337:85 , srv1819221209:85 , srv1849811035:85 , srv2074320680:85 , srv261689959:85 , srv282062910:85 , srv296797387:85 , srv425250220:85 , srv589944039:85 , srv638296674:85 , srv678097564:85 , srv715627078:85 , srv853148212:85 , srv874605768:85 , srv877565223:85 , srv1094003071:86 , srv1178467675:86 , srv1205079912:86 , srv1211983039:86 , srv1234839184:86 , srv1370731162:86 , srv1496597095:86 , srv1667533585:86 , srv1702902910:86 , srv1709880603:86 , srv1892285893:86 , srv2025185998:86 , srv2085917343:86 , srv234951261:86 , srv27723583:86 , srv287168230:86 , srv429443379:86 , srv605116776:86 , srv642203643:86 , srv922413485:86 , srv98165554:86 , srv1031569012:87 , srv1059143689:87 , srv1342331011:87 , srv1360493600:87 , srv1376730872:87 , srv1391941215:87 , srv1397304305:87 , srv1636683345:87 , srv1664138818:87 , srv1740590547:87 , srv2084930318:87 , srv2090839365:87 , srv2141584619:87 , srv341325817:87 , srv475379161:87 , srv53516150:87 , srv847123025:87 , srv1557894940:88 , srv1614137333:88 , srv1870386760:88 , srv1929274594:88 , srv25100633:88 , srv510101192:88 , srv672401494:88 , srv896657461:88 , srv1879399424:89 , srv1236328696:560 } 2023-04-28 18:55:47,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-28 18:55:47,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv672401494=86, srv208385227=53, srv212027338=57, srv341325817=68, srv877565223=96, srv1202839180=13, srv1001874292=0, srv287168230=65, srv1710946337=38, srv1167752309=10, srv2025185998=49, srv475379161=75, srv102600028=1, srv678097564=87, srv1220013637=16, srv1473702985=25, srv896657461=97, srv1867284163=42, srv847123025=93, srv1612378940=30, srv1702902910=36, srv922413485=98, srv1664138818=34, srv1570256225=28, srv2084930318=54, srv282062910=64, srv1035115816=3, srv1819221209=40, srv19890972=48, srv699291259=88, srv799381491=91, srv1234839184=17, srv1391941215=23, srv332174597=67, srv1740590547=39, srv1370731162=21, srv1376730872=22, srv1709880603=37, srv2090839365=56, srv1031569012=2, srv2074320680=52, srv1342331011=19, srv25100633=61, srv715627078=90, srv1360493600=20, srv98165554=99, srv1879399424=45, srv296797387=66, srv510101192=77, srv27723583=63, srv1129187718=9, srv1176044827=11, srv638296674=83, srv1849811035=41, srv2145214720=59, srv639392223=84, srv874605768=95, srv1496597095=26, srv2085917343=55, srv429443379=72, srv44548987=73, srv589944039=80, srv234951261=60, srv1878589687=44, srv1636683345=33, srv1870386760=43, srv2033130284=50, srv2141584619=58, srv396545723=69, srv476867875=76, srv600568230=81, srv1557894940=27, srv826291460=92, srv461111322=74, srv397250822=70, srv1122887864=8, srv642203643=85, srv1634907019=32, srv1929274594=47, srv1205079912=14, srv1614137333=31, srv605116776=82, srv261689959=62, srv1178467675=12, srv1397304305=24, srv1236328696=18, srv709688627=89, srv2072817726=51, srv529002561=78, srv1211983039=15, srv1108549330=7, srv53516150=79, srv1667533585=35, srv1059143689=4, srv1593942592=29, srv1094003071=6, srv1892285893=46, srv853148212=94, srv1080975212=5, srv425250220=71} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-28 18:55:47,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-28 18:55:47,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-28 18:55:47,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-28 18:55:47,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-28 18:55:47,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-28 18:55:47,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-28 18:55:47,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-28 18:55:47,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-28 18:55:47,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-28 18:55:47,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-28 18:55:47,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-28 18:55:47,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-28 18:55:47,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-28 18:55:47,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-28 18:55:47,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-28 18:55:47,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-28 18:55:47,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-28 18:55:47,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-28 18:55:47,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-28 18:55:47,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-28 18:55:47,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-28 18:55:47,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-28 18:55:47,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-28 18:55:47,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-28 18:55:47,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-28 18:55:47,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-28 18:55:47,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-28 18:55:47,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-28 18:55:47,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-28 18:55:47,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-28 18:55:47,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-28 18:55:47,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-28 18:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-28 18:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-28 18:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-28 18:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-28 18:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-28 18:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-28 18:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-28 18:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-28 18:55:47,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-28 18:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-28 18:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-28 18:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-28 18:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-28 18:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-28 18:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-28 18:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-28 18:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-28 18:55:47,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-28 18:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-28 18:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-28 18:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-28 18:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-28 18:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-28 18:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-28 18:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-28 18:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-28 18:55:47,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-28 18:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-28 18:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-28 18:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-28 18:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-28 18:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-28 18:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-28 18:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-28 18:55:47,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-28 18:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-28 18:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-28 18:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-28 18:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-28 18:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-28 18:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-28 18:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-28 18:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-28 18:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-28 18:55:47,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-28 18:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-28 18:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-28 18:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-28 18:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-28 18:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-28 18:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-28 18:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-28 18:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-28 18:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-28 18:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-28 18:55:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-28 18:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-28 18:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-28 18:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-28 18:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-28 18:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-28 18:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-28 18:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-28 18:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-28 18:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-28 18:55:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-28 18:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-28 18:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-28 18:55:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-28 18:55:47,939 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006122578771996661); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0089643263564985); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006122578771996661); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:55:47,939 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-28 18:55:47,939 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.001142670485230373, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006122578771996661); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0089643263564985); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006122578771996661); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-28 18:55:48,353 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 550 ms to try 5000 different iterations. Found a solution that moves 257 regions; Going from a computed imbalance of 0.001142670485230373 to a new imbalance of 2.7864766154058903E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.028555555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:55:48,355 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1870386760:82 , srv1031569012:83 , srv1376730872:83 , srv1557894940:83 , srv1667533585:83 , srv1879399424:83 , srv425250220:83 , srv600568230:83 , srv672401494:83 , srv877565223:83 , srv1059143689:84 , srv1167752309:84 , srv1202839180:84 , srv1342331011:84 , srv1360493600:84 , srv1370731162:84 , srv1391941215:84 , srv1397304305:84 , srv1496597095:84 , srv1593942592:84 , srv1614137333:84 , srv1702902910:84 , srv1709880603:84 , srv1740590547:84 , srv1892285893:84 , srv1929274594:84 , srv2085917343:84 , srv2141584619:84 , srv234951261:84 , srv25100633:84 , srv27723583:84 , srv282062910:84 , srv287168230:84 , srv510101192:84 , srv53516150:84 , srv589944039:84 , srv639392223:84 , srv699291259:84 , srv847123025:84 , srv853148212:84 , srv874605768:84 , srv1001874292:85 , srv1035115816:85 , srv1080975212:85 , srv1094003071:85 , srv1108549330:85 , srv1122887864:85 , srv1178467675:85 , srv1205079912:85 , srv1211983039:85 , srv1220013637:85 , srv1234839184:85 , srv1570256225:85 , srv1612378940:85 , srv1636683345:85 , srv1664138818:85 , srv1710946337:85 , srv1819221209:85 , srv1849811035:85 , srv1867284163:85 , srv1878589687:85 , srv19890972:85 , srv2025185998:85 , srv2033130284:85 , srv2072817726:85 , srv2074320680:85 , srv2084930318:85 , srv2090839365:85 , srv212027338:85 , srv2145214720:85 , srv261689959:85 , srv332174597:85 , srv341325817:85 , srv396545723:85 , srv397250822:85 , srv429443379:85 , srv44548987:85 , srv461111322:85 , srv475379161:85 , srv476867875:85 , srv529002561:85 , srv605116776:85 , srv638296674:85 , srv642203643:85 , srv678097564:85 , srv709688627:85 , srv715627078:85 , srv799381491:85 , srv826291460:85 , srv896657461:85 , srv922413485:85 , srv98165554:85 , srv102600028:86 , srv1129187718:86 , srv1176044827:86 , srv1473702985:86 , srv1634907019:86 , srv208385227:86 , srv296797387:86 , srv1236328696:630 } 2023-04-28 18:55:48,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-28 18:55:48,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv672401494=86, srv208385227=53, srv212027338=57, srv341325817=68, srv877565223=96, srv1202839180=13, srv1001874292=0, srv287168230=65, srv1710946337=38, srv1167752309=10, srv2025185998=49, srv475379161=75, srv102600028=1, srv678097564=87, srv1220013637=16, srv1473702985=25, srv896657461=97, srv1867284163=42, srv847123025=93, srv1612378940=30, srv1702902910=36, srv922413485=98, srv1664138818=34, srv1570256225=28, srv2084930318=54, srv282062910=64, srv1035115816=3, srv1819221209=40, srv19890972=48, srv699291259=88, srv799381491=91, srv1234839184=17, srv1391941215=23, srv332174597=67, srv1740590547=39, srv1370731162=21, srv1376730872=22, srv1709880603=37, srv2090839365=56, srv1031569012=2, srv2074320680=52, srv1342331011=19, srv25100633=61, srv715627078=90, srv1360493600=20, srv98165554=99, srv1879399424=45, srv296797387=66, srv510101192=77, srv27723583=63, srv1129187718=9, srv1176044827=11, srv638296674=83, srv1849811035=41, srv2145214720=59, srv639392223=84, srv874605768=95, srv1496597095=26, srv2085917343=55, srv429443379=72, srv44548987=73, srv589944039=80, srv234951261=60, srv1878589687=44, srv1636683345=33, srv1870386760=43, srv2033130284=50, srv2141584619=58, srv396545723=69, srv476867875=76, srv600568230=81, srv1557894940=27, srv826291460=92, srv461111322=74, srv397250822=70, srv1122887864=8, srv642203643=85, srv1634907019=32, srv1929274594=47, srv1205079912=14, srv1614137333=31, srv605116776=82, srv261689959=62, srv1178467675=12, srv1397304305=24, srv1236328696=18, srv709688627=89, srv2072817726=51, srv529002561=78, srv1211983039=15, srv1108549330=7, srv53516150=79, srv1667533585=35, srv1059143689=4, srv1593942592=29, srv1094003071=6, srv1892285893=46, srv853148212=94, srv1080975212=5, srv425250220=71} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-28 18:55:48,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-28 18:55:48,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-28 18:55:48,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-28 18:55:48,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-28 18:55:48,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-28 18:55:48,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-28 18:55:48,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-28 18:55:48,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-28 18:55:48,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-28 18:55:48,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-28 18:55:48,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-28 18:55:48,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-28 18:55:48,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-28 18:55:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-28 18:55:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-28 18:55:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-28 18:55:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-28 18:55:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-28 18:55:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-28 18:55:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-28 18:55:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-28 18:55:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-28 18:55:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-28 18:55:48,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-28 18:55:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-28 18:55:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-28 18:55:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-28 18:55:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-28 18:55:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-28 18:55:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-28 18:55:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-28 18:55:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-28 18:55:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-28 18:55:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-28 18:55:48,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-28 18:55:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-28 18:55:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-28 18:55:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-28 18:55:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-28 18:55:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-28 18:55:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-28 18:55:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-28 18:55:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-28 18:55:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-28 18:55:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-28 18:55:48,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-28 18:55:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-28 18:55:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-28 18:55:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-28 18:55:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-28 18:55:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-28 18:55:48,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-28 18:55:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-28 18:55:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-28 18:55:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-28 18:55:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-28 18:55:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-28 18:55:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-28 18:55:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-28 18:55:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-28 18:55:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-28 18:55:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-28 18:55:48,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-28 18:55:48,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-28 18:55:48,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-28 18:55:48,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-28 18:55:48,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-28 18:55:48,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-28 18:55:48,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-28 18:55:48,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-28 18:55:48,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-28 18:55:48,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-28 18:55:48,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-28 18:55:48,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-28 18:55:48,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-28 18:55:48,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-28 18:55:48,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-28 18:55:48,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-28 18:55:48,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-28 18:55:48,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-28 18:55:48,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-28 18:55:48,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-28 18:55:48,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-28 18:55:48,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-28 18:55:48,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-28 18:55:48,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-28 18:55:48,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-28 18:55:48,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-28 18:55:48,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-28 18:55:48,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-28 18:55:48,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-28 18:55:48,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-28 18:55:48,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-28 18:55:48,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-28 18:55:48,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-28 18:55:48,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-28 18:55:48,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-28 18:55:48,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-28 18:55:48,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-28 18:55:48,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-28 18:55:48,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-28 18:55:48,529 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:55:48,529 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-28 18:55:48,530 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6065213691452885E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-28 18:55:48,843 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 434 ms to try 5000 different iterations. Found a solution that moves 20 regions; Going from a computed imbalance of 2.6065213691452885E-5 to a new imbalance of 8.1197331102395E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0022222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:55:48,843 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1129187718:83 , srv1176044827:83 , srv1211983039:83 , srv1234839184:83 , srv1391941215:83 , srv1634907019:83 , srv678097564:83 , srv1001874292:84 , srv102600028:84 , srv1031569012:84 , srv1035115816:84 , srv1059143689:84 , srv1080975212:84 , srv1094003071:84 , srv1108549330:84 , srv1122887864:84 , srv1178467675:84 , srv1202839180:84 , srv1205079912:84 , srv1220013637:84 , srv1342331011:84 , srv1360493600:84 , srv1370731162:84 , srv1376730872:84 , srv1397304305:84 , srv1473702985:84 , srv1557894940:84 , srv1570256225:84 , srv1593942592:84 , srv1612378940:84 , srv1614137333:84 , srv1636683345:84 , srv1664138818:84 , srv1667533585:84 , srv1702902910:84 , srv1709880603:84 , srv1740590547:84 , srv1819221209:84 , srv1849811035:84 , srv1867284163:84 , srv1870386760:84 , srv1878589687:84 , srv1879399424:84 , srv1929274594:84 , srv19890972:84 , srv2025185998:84 , srv2033130284:84 , srv2072817726:84 , srv2074320680:84 , srv208385227:84 , srv2084930318:84 , srv2085917343:84 , srv2090839365:84 , srv212027338:84 , srv2141584619:84 , srv2145214720:84 , srv234951261:84 , srv25100633:84 , srv261689959:84 , srv27723583:84 , srv282062910:84 , srv296797387:84 , srv332174597:84 , srv341325817:84 , srv396545723:84 , srv397250822:84 , srv429443379:84 , srv44548987:84 , srv461111322:84 , srv476867875:84 , srv510101192:84 , srv529002561:84 , srv53516150:84 , srv589944039:84 , srv600568230:84 , srv605116776:84 , srv638296674:84 , srv639392223:84 , srv642203643:84 , srv672401494:84 , srv699291259:84 , srv709688627:84 , srv715627078:84 , srv799381491:84 , srv826291460:84 , srv847123025:84 , srv853148212:84 , srv874605768:84 , srv877565223:84 , srv896657461:84 , srv922413485:84 , srv98165554:84 , srv1167752309:85 , srv1496597095:85 , srv1710946337:85 , srv1892285893:85 , srv287168230:85 , srv425250220:85 , srv475379161:85 , srv1236328696:684 } 2023-04-28 18:55:48,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-28 18:55:48,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv672401494=86, srv208385227=53, srv212027338=57, srv341325817=68, srv877565223=96, srv1202839180=13, srv1001874292=0, srv287168230=65, srv1710946337=38, srv1167752309=10, srv2025185998=49, srv475379161=75, srv102600028=1, srv678097564=87, srv1220013637=16, srv1473702985=25, srv896657461=97, srv1867284163=42, srv847123025=93, srv1612378940=30, srv1702902910=36, srv922413485=98, srv1664138818=34, srv1570256225=28, srv2084930318=54, srv282062910=64, srv1035115816=3, srv1819221209=40, srv19890972=48, srv699291259=88, srv799381491=91, srv1234839184=17, srv1391941215=23, srv332174597=67, srv1740590547=39, srv1370731162=21, srv1376730872=22, srv1709880603=37, srv2090839365=56, srv1031569012=2, srv2074320680=52, srv1342331011=19, srv25100633=61, srv715627078=90, srv1360493600=20, srv98165554=99, srv1879399424=45, srv296797387=66, srv510101192=77, srv27723583=63, srv1129187718=9, srv1176044827=11, srv638296674=83, srv1849811035=41, srv2145214720=59, srv639392223=84, srv874605768=95, srv1496597095=26, srv2085917343=55, srv429443379=72, srv44548987=73, srv589944039=80, srv234951261=60, srv1878589687=44, srv1636683345=33, srv1870386760=43, srv2033130284=50, srv2141584619=58, srv396545723=69, srv476867875=76, srv600568230=81, srv1557894940=27, srv826291460=92, srv461111322=74, srv397250822=70, srv1122887864=8, srv642203643=85, srv1634907019=32, srv1929274594=47, srv1205079912=14, srv1614137333=31, srv605116776=82, srv261689959=62, srv1178467675=12, srv1397304305=24, srv1236328696=18, srv709688627=89, srv2072817726=51, srv529002561=78, srv1211983039=15, srv1108549330=7, srv53516150=79, srv1667533585=35, srv1059143689=4, srv1593942592=29, srv1094003071=6, srv1892285893=46, srv853148212=94, srv1080975212=5, srv425250220=71} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-28 18:55:48,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-28 18:55:48,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-28 18:55:48,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-28 18:55:48,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-28 18:55:48,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-28 18:55:48,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-28 18:55:48,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-28 18:55:48,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-28 18:55:48,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-28 18:55:48,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-28 18:55:48,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-28 18:55:48,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-28 18:55:48,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-28 18:55:48,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-28 18:55:48,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-28 18:55:48,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-28 18:55:48,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-28 18:55:48,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-28 18:55:48,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-28 18:55:48,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-28 18:55:48,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-28 18:55:48,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-28 18:55:48,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-28 18:55:48,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-28 18:55:48,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-28 18:55:48,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-28 18:55:48,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-28 18:55:48,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-28 18:55:48,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-28 18:55:48,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-28 18:55:48,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-28 18:55:48,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-28 18:55:48,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-28 18:55:48,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-28 18:55:48,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-28 18:55:48,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-28 18:55:48,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-28 18:55:48,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-28 18:55:48,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-28 18:55:48,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-28 18:55:48,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-28 18:55:48,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-28 18:55:48,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-28 18:55:48,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-28 18:55:48,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-28 18:55:48,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-28 18:55:48,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-28 18:55:48,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-28 18:55:48,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-28 18:55:48,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-28 18:55:48,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-28 18:55:48,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-28 18:55:48,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-28 18:55:48,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-28 18:55:48,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-28 18:55:48,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-28 18:55:48,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-28 18:55:48,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-28 18:55:48,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-28 18:55:48,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-28 18:55:48,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-28 18:55:48,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-28 18:55:48,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-28 18:55:48,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-28 18:55:48,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-28 18:55:48,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-28 18:55:48,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-28 18:55:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-28 18:55:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-28 18:55:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-28 18:55:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-28 18:55:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-28 18:55:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-28 18:55:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-28 18:55:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-28 18:55:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-28 18:55:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-28 18:55:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-28 18:55:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-28 18:55:48,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-28 18:55:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-28 18:55:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-28 18:55:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-28 18:55:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-28 18:55:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-28 18:55:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-28 18:55:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-28 18:55:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-28 18:55:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-28 18:55:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-28 18:55:48,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-28 18:55:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-28 18:55:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-28 18:55:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-28 18:55:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-28 18:55:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-28 18:55:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-28 18:55:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-28 18:55:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-28 18:55:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-28 18:55:48,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-28 18:55:49,007 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.979690117040589E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:55:49,007 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1129187718:83 , srv1176044827:83 , srv1211983039:83 , srv1234839184:83 , srv1391941215:83 , srv1634907019:83 , srv678097564:83 , srv1001874292:84 , srv102600028:84 , srv1031569012:84 , srv1035115816:84 , srv1059143689:84 , srv1080975212:84 , srv1094003071:84 , srv1108549330:84 , srv1122887864:84 , srv1178467675:84 , srv1202839180:84 , srv1205079912:84 , srv1220013637:84 , srv1342331011:84 , srv1360493600:84 , srv1370731162:84 , srv1376730872:84 , srv1397304305:84 , srv1473702985:84 , srv1557894940:84 , srv1570256225:84 , srv1593942592:84 , srv1612378940:84 , srv1614137333:84 , srv1636683345:84 , srv1664138818:84 , srv1667533585:84 , srv1702902910:84 , srv1709880603:84 , srv1740590547:84 , srv1819221209:84 , srv1849811035:84 , srv1867284163:84 , srv1870386760:84 , srv1878589687:84 , srv1879399424:84 , srv1929274594:84 , srv19890972:84 , srv2025185998:84 , srv2033130284:84 , srv2072817726:84 , srv2074320680:84 , srv208385227:84 , srv2084930318:84 , srv2085917343:84 , srv2090839365:84 , srv212027338:84 , srv2141584619:84 , srv2145214720:84 , srv234951261:84 , srv25100633:84 , srv261689959:84 , srv27723583:84 , srv282062910:84 , srv296797387:84 , srv332174597:84 , srv341325817:84 , srv396545723:84 , srv397250822:84 , srv429443379:84 , srv44548987:84 , srv461111322:84 , srv476867875:84 , srv510101192:84 , srv529002561:84 , srv53516150:84 , srv589944039:84 , srv600568230:84 , srv605116776:84 , srv638296674:84 , srv639392223:84 , srv642203643:84 , srv672401494:84 , srv699291259:84 , srv709688627:84 , srv715627078:84 , srv799381491:84 , srv826291460:84 , srv847123025:84 , srv853148212:84 , srv874605768:84 , srv877565223:84 , srv896657461:84 , srv922413485:84 , srv98165554:84 , srv1167752309:85 , srv1496597095:85 , srv1710946337:85 , srv1892285893:85 , srv287168230:85 , srv425250220:85 , srv475379161:85 , srv1236328696:684 } 2023-04-28 18:55:49,047 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=417 (was 401) - SystemLoadAverage LEAK? -, ProcessCount=177 (was 176) - ProcessCount LEAK? -, AvailableMemoryMB=12042 (was 12547) 2023-04-28 18:55:49,064 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=417, ProcessCount=177, AvailableMemoryMB=12042 2023-04-28 18:55:49,065 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-28 18:55:49,065 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-04-28 18:55:49,080 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1017592761:84 , srv10639762:84 , srv1134992552:84 , srv1253221099:84 , srv1307913741:84 , srv1327549255:84 , srv1357373047:84 , srv1372842337:84 , srv1390725352:84 , srv139894299:84 , srv1419695896:84 , srv1436603787:84 , srv1483086408:84 , srv1612557299:84 , srv1651468315:84 , srv2108487641:84 , srv2139035017:84 , srv265795151:84 , srv33093390:84 , srv333840063:84 , srv375190710:84 , srv49551222:84 , srv754814469:84 , srv763171625:84 , srv863215366:84 , srv874137566:84 , srv878571160:84 , srv901842393:84 , srv945145572:84 , srv181954346:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-04-28 18:55:49,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-28 18:55:49,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1253221099=3, srv181954346=15, srv1651468315=14, srv1307913741=4, srv863215366=25, srv375190710=21, srv2108487641=16, srv33093390=19, srv1134992552=2, srv1419695896=10, srv1390725352=8, srv49551222=22, srv754814469=23, srv1327549255=5, srv139894299=9, srv874137566=26, srv763171625=24, srv1017592761=0, srv10639762=1, srv2139035017=17, srv333840063=20, srv265795151=18, srv1483086408=12, srv1357373047=6, srv901842393=28, srv945145572=29, srv1372842337=7, srv878571160=27, srv1612557299=13, srv1436603787=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-28 18:55:49,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-28 18:55:49,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-28 18:55:49,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-28 18:55:49,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-28 18:55:49,124 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0863679120300245, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:55:49,125 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912465185734916, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0863679120300245, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-04-28 18:57:49,103 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6109964 different iterations. Found a solution that moves 1858 regions; Going from a computed imbalance of 0.9912465185734916 to a new imbalance of 4.336664689392977E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6881481481481482, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:57:49,110 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1017592761:90 , srv10639762:90 , srv1134992552:90 , srv1253221099:90 , srv1307913741:90 , srv1327549255:90 , srv1357373047:90 , srv1372842337:90 , srv1390725352:90 , srv139894299:90 , srv1419695896:90 , srv1436603787:90 , srv1483086408:90 , srv1612557299:90 , srv1651468315:90 , srv181954346:90 , srv2108487641:90 , srv2139035017:90 , srv265795151:90 , srv33093390:90 , srv333840063:90 , srv375190710:90 , srv49551222:90 , srv754814469:90 , srv763171625:90 , srv863215366:90 , srv874137566:90 , srv878571160:90 , srv901842393:90 , srv945145572:90 } 2023-04-28 18:57:49,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-28 18:57:49,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1253221099=3, srv181954346=15, srv1651468315=14, srv1307913741=4, srv863215366=25, srv375190710=21, srv2108487641=16, srv33093390=19, srv1134992552=2, srv1419695896=10, srv1390725352=8, srv49551222=22, srv754814469=23, srv1327549255=5, srv139894299=9, srv874137566=26, srv763171625=24, srv1017592761=0, srv10639762=1, srv2139035017=17, srv333840063=20, srv265795151=18, srv1483086408=12, srv1357373047=6, srv901842393=28, srv945145572=29, srv1372842337=7, srv878571160=27, srv1612557299=13, srv1436603787=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-28 18:57:49,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-28 18:57:49,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-28 18:57:49,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-28 18:57:49,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-28 18:57:49,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-28 18:57:49,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-28 18:57:49,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-28 18:57:49,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-28 18:57:49,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-28 18:57:49,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-28 18:57:49,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-28 18:57:49,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-28 18:57:49,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-28 18:57:49,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-28 18:57:49,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-28 18:57:49,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-28 18:57:49,135 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-28 18:57:49,136 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1017592761:90 , srv10639762:90 , srv1134992552:90 , srv1253221099:90 , srv1307913741:90 , srv1327549255:90 , srv1357373047:90 , srv1372842337:90 , srv1390725352:90 , srv139894299:90 , srv1419695896:90 , srv1436603787:90 , srv1483086408:90 , srv1612557299:90 , srv1651468315:90 , srv181954346:90 , srv2108487641:90 , srv2139035017:90 , srv265795151:90 , srv33093390:90 , srv333840063:90 , srv375190710:90 , srv49551222:90 , srv754814469:90 , srv763171625:90 , srv863215366:90 , srv874137566:90 , srv878571160:90 , srv901842393:90 , srv945145572:90 } 2023-04-28 18:57:49,153 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=300 (was 417), ProcessCount=179 (was 177) - ProcessCount LEAK? -, AvailableMemoryMB=11794 (was 12042)