2024-02-11 05:53:12,230 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-02-11 05:53:12,543 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-02-11 05:53:12,642 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-02-11 05:53:12,704 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-11 05:53:12,711 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-11 05:53:12,725 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=224, ProcessCount=170, AvailableMemoryMB=3159 2024-02-11 05:53:12,727 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-11 05:53:12,727 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-11 05:53:12,866 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1034226518:84 , srv1039266885:84 , srv1043126507:84 , srv1069773820:84 , srv1116486723:84 , srv1137559323:84 , srv1141373266:84 , srv1144844548:84 , srv1154154547:84 , srv1154554301:84 , srv1195637595:84 , srv1206844611:84 , srv1274807635:84 , srv1281395006:84 , srv1332241794:84 , srv1339063797:84 , srv1355822952:84 , srv137177329:84 , srv1373617352:84 , srv1383916290:84 , srv1385560946:84 , srv1386728933:84 , srv1405758175:84 , srv1407743990:84 , srv1409806377:84 , srv1410261826:84 , srv1411605209:84 , srv1417038201:84 , srv1429471019:84 , srv1434621581:84 , srv145229301:84 , srv1491816555:84 , srv1509559165:84 , srv155773995:84 , srv1567449060:84 , srv1569704449:84 , srv159956322:84 , srv1605371865:84 , srv1615693801:84 , srv1650470769:84 , srv1653374971:84 , srv1692003641:84 , srv171382807:84 , srv1725417758:84 , srv1726518293:84 , srv1760818363:84 , srv1783985405:84 , srv1788974272:84 , srv1791605415:84 , srv1799191199:84 , srv1805117166:84 , srv1821429752:84 , srv1872833825:84 , srv1878116048:84 , srv1916485313:84 , srv1960010711:84 , srv1963688432:84 , srv1973662389:84 , srv2008079088:84 , srv2039541758:84 , srv2110072049:84 , srv230140836:84 , srv232642562:84 , srv238745902:84 , srv243443735:84 , srv28694002:84 , srv304460622:84 , srv343411892:84 , srv346085983:84 , srv360636389:84 , srv374904780:84 , srv376976694:84 , srv384215703:84 , srv393279496:84 , srv401882966:84 , srv419049197:84 , srv422444611:84 , srv478038710:84 , srv486440582:84 , srv532447706:84 , srv54488016:84 , srv568165290:84 , srv646974177:84 , srv667381576:84 , srv66935700:84 , srv694342788:84 , srv699832196:84 , srv726065041:84 , srv747159586:84 , srv751092234:84 , srv776132788:84 , srv812841426:84 , srv827317314:84 , srv883293631:84 , srv911448119:84 , srv938782263:84 , srv944719594:84 , srv948344341:84 , srv98004355:84 , srv127162015:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-02-11 05:53:12,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 05:53:12,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2110072049=61, srv1791605415=49, srv1405758175=23, srv667381576=84, srv948344341=98, srv1154154547=8, srv98004355=99, srv1144844548=7, srv171382807=43, srv1821429752=52, srv1116486723=4, srv1069773820=3, srv1434621581=30, srv1760818363=46, srv1963688432=57, srv238745902=64, srv1783985405=47, srv812841426=92, srv694342788=86, srv911448119=95, srv1726518293=45, srv486440582=79, srv346085983=69, srv776132788=91, srv1417038201=28, srv155773995=34, srv1650470769=40, srv1692003641=42, srv1034226518=0, srv568165290=82, srv1195637595=10, srv159956322=37, srv1332241794=15, srv944719594=97, srv1137559323=5, srv393279496=74, srv1805117166=51, srv1043126507=2, srv1491816555=32, srv137177329=18, srv1788974272=48, srv2008079088=59, srv1039266885=1, srv1154554301=9, srv726065041=88, srv1385560946=21, srv1653374971=41, srv883293631=94, srv304460622=67, srv1429471019=29, srv1799191199=50, srv1960010711=56, srv243443735=65, srv127162015=12, srv532447706=80, srv1509559165=33, srv343411892=68, srv145229301=31, srv699832196=87, srv1567449060=35, srv747159586=89, srv1141373266=6, srv1373617352=19, srv1274807635=13, srv1407743990=24, srv1725417758=44, srv230140836=62, srv384215703=73, srv1281395006=14, srv1383916290=20, srv54488016=81, srv827317314=93, srv376976694=72, srv1605371865=38, srv646974177=83, srv66935700=85, srv360636389=70, srv1386728933=22, srv1569704449=36, srv1916485313=55, srv28694002=66, srv374904780=71, srv401882966=75, srv2039541758=60, srv1878116048=54, srv1355822952=17, srv1410261826=26, srv1973662389=58, srv419049197=76, srv938782263=96, srv751092234=90, srv1206844611=11, srv1409806377=25, srv1872833825=53, srv478038710=78, srv1339063797=16, srv1615693801=39, srv422444611=77, srv232642562=63, srv1411605209=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 05:53:12,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 05:53:12,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 05:53:12,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 05:53:12,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 05:53:12,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 05:53:12,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 05:53:12,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 05:53:12,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 05:53:12,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 05:53:12,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 05:53:12,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 05:53:12,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 05:53:12,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 05:53:12,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 05:53:12,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 05:53:12,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 05:53:12,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 05:53:12,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 05:53:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 05:53:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 05:53:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 05:53:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 05:53:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 05:53:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 05:53:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 05:53:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 05:53:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 05:53:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 05:53:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 05:53:12,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 05:53:12,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 05:53:12,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 05:53:12,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 05:53:12,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 05:53:12,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 05:53:12,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 05:53:12,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 05:53:12,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 05:53:12,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 05:53:12,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 05:53:12,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 05:53:12,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 05:53:12,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 05:53:12,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 05:53:12,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 05:53:12,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 05:53:12,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 05:53:12,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 05:53:12,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 05:53:12,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 05:53:12,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 05:53:12,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 05:53:12,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 05:53:12,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 05:53:12,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 05:53:12,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 05:53:12,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 05:53:12,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 05:53:12,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 05:53:12,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 05:53:12,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 05:53:12,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 05:53:12,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 05:53:12,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 05:53:12,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 05:53:12,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 05:53:12,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 05:53:12,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 05:53:12,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 05:53:12,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 05:53:12,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 05:53:12,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 05:53:12,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 05:53:12,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 05:53:12,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 05:53:12,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 05:53:12,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 05:53:12,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 05:53:12,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 05:53:12,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 05:53:12,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 05:53:12,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 05:53:12,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 05:53:12,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 05:53:12,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 05:53:12,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 05:53:12,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 05:53:12,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 05:53:12,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 05:53:12,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 05:53:12,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 05:53:12,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 05:53:12,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 05:53:12,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 05:53:12,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 05:53:12,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 05:53:12,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 05:53:12,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 05:53:12,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 05:53:12,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 05:53:12,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 05:53:13,227 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:53:13,227 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 05:53:13,228 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 05:53:13,568 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 555 ms to try 5000 different iterations. Found a solution that moves 4707 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18408502614002906. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0426416055280547, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05727637919698614, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.523, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0426416055280547, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17908333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:53:13,579 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1137559323:80 , srv1405758175:80 , srv1434621581:81 , srv1195637595:82 , srv2008079088:82 , srv1373617352:83 , srv1726518293:83 , srv944719594:83 , srv98004355:83 , srv1141373266:84 , srv1386728933:84 , srv1650470769:84 , srv171382807:84 , srv1821429752:84 , srv1872833825:84 , srv2039541758:84 , srv2110072049:84 , srv346085983:84 , srv384215703:84 , srv422444611:84 , srv478038710:84 , srv699832196:84 , srv948344341:84 , srv1154154547:85 , srv137177329:85 , srv1407743990:85 , srv1410261826:85 , srv1653374971:85 , srv1692003641:85 , srv1725417758:85 , srv1788974272:85 , srv1805117166:85 , srv238745902:85 , srv243443735:85 , srv304460622:85 , srv374904780:85 , srv646974177:85 , srv747159586:85 , srv812841426:85 , srv911448119:85 , srv1039266885:86 , srv1043126507:86 , srv1281395006:86 , srv1417038201:86 , srv1429471019:86 , srv1567449060:86 , srv1615693801:86 , srv1783985405:86 , srv1878116048:86 , srv1960010711:86 , srv1963688432:86 , srv1973662389:86 , srv230140836:86 , srv486440582:86 , srv54488016:86 , srv667381576:86 , srv694342788:86 , srv751092234:86 , srv938782263:86 , srv1034226518:87 , srv1069773820:87 , srv1154554301:87 , srv1332241794:87 , srv1355822952:87 , srv1409806377:87 , srv145229301:87 , srv1491816555:87 , srv1605371865:87 , srv1799191199:87 , srv1916485313:87 , srv343411892:87 , srv360636389:87 , srv419049197:87 , srv568165290:87 , srv827317314:87 , srv1206844611:88 , srv1274807635:88 , srv1411605209:88 , srv1509559165:88 , srv159956322:88 , srv1760818363:88 , srv1791605415:88 , srv393279496:88 , srv401882966:88 , srv532447706:88 , srv66935700:88 , srv726065041:88 , srv776132788:88 , srv1144844548:89 , srv1339063797:89 , srv1385560946:89 , srv155773995:89 , srv1569704449:90 , srv883293631:90 , srv1383916290:91 , srv28694002:91 , srv232642562:92 , srv376976694:92 , srv1116486723:101 , srv127162015:469 } 2024-02-11 05:53:13,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 05:53:13,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2110072049=61, srv1791605415=49, srv1405758175=23, srv667381576=84, srv948344341=98, srv1154154547=8, srv98004355=99, srv1144844548=7, srv171382807=43, srv1821429752=52, srv1116486723=4, srv1069773820=3, srv1434621581=30, srv1760818363=46, srv1963688432=57, srv238745902=64, srv1783985405=47, srv812841426=92, srv694342788=86, srv911448119=95, srv1726518293=45, srv486440582=79, srv346085983=69, srv776132788=91, srv1417038201=28, srv155773995=34, srv1650470769=40, srv1692003641=42, srv1034226518=0, srv568165290=82, srv1195637595=10, srv159956322=37, srv1332241794=15, srv944719594=97, srv1137559323=5, srv393279496=74, srv1805117166=51, srv1043126507=2, srv1491816555=32, srv137177329=18, srv1788974272=48, srv2008079088=59, srv1039266885=1, srv1154554301=9, srv726065041=88, srv1385560946=21, srv1653374971=41, srv883293631=94, srv304460622=67, srv1429471019=29, srv1799191199=50, srv1960010711=56, srv243443735=65, srv127162015=12, srv532447706=80, srv1509559165=33, srv343411892=68, srv145229301=31, srv699832196=87, srv1567449060=35, srv747159586=89, srv1141373266=6, srv1373617352=19, srv1274807635=13, srv1407743990=24, srv1725417758=44, srv230140836=62, srv384215703=73, srv1281395006=14, srv1383916290=20, srv54488016=81, srv827317314=93, srv376976694=72, srv1605371865=38, srv646974177=83, srv66935700=85, srv360636389=70, srv1386728933=22, srv1569704449=36, srv1916485313=55, srv28694002=66, srv374904780=71, srv401882966=75, srv2039541758=60, srv1878116048=54, srv1355822952=17, srv1410261826=26, srv1973662389=58, srv419049197=76, srv938782263=96, srv751092234=90, srv1206844611=11, srv1409806377=25, srv1872833825=53, srv478038710=78, srv1339063797=16, srv1615693801=39, srv422444611=77, srv232642562=63, srv1411605209=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 05:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 05:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 05:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 05:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 05:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 05:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 05:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 05:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 05:53:13,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 05:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 05:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 05:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 05:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 05:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 05:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 05:53:13,599 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 05:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 05:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 05:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 05:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 05:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 05:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 05:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 05:53:13,600 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 05:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 05:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 05:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 05:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 05:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 05:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 05:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 05:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 05:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 05:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 05:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 05:53:13,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 05:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 05:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 05:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 05:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 05:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 05:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 05:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 05:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 05:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 05:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 05:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 05:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 05:53:13,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 05:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 05:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 05:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 05:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 05:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 05:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 05:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 05:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 05:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 05:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 05:53:13,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 05:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 05:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 05:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 05:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 05:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 05:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 05:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 05:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 05:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 05:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 05:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 05:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 05:53:13,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 05:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 05:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 05:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 05:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 05:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 05:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 05:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 05:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 05:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 05:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 05:53:13,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 05:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 05:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 05:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 05:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 05:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 05:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 05:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 05:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 05:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 05:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 05:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 05:53:13,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 05:53:13,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 05:53:13,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 05:53:13,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 05:53:13,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 05:53:13,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 05:53:13,759 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0426416055280547, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05727637919698614, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0426416055280547, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17908333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:53:13,760 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 05:53:13,760 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1840520670215797, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0426416055280547, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05727637919698614, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0426416055280547, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17908333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 05:53:14,079 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 447 ms to try 5000 different iterations. Found a solution that moves 2659 regions; Going from a computed imbalance of 0.1840520670215797 to a new imbalance of 0.016496044712995653. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023740501740078745); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03679042475339682, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29544444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023740501740078745); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:53:14,100 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1116486723:73 , srv232642562:79 , srv376976694:79 , srv155773995:81 , srv28694002:81 , srv1144844548:82 , srv1154554301:82 , srv776132788:82 , srv1274807635:83 , srv1281395006:83 , srv1339063797:83 , srv1385560946:83 , srv1569704449:83 , srv159956322:83 , srv1799191199:83 , srv343411892:83 , srv66935700:83 , srv751092234:83 , srv883293631:83 , srv938782263:83 , srv1043126507:84 , srv1332241794:84 , srv1491816555:84 , srv1783985405:84 , srv1878116048:84 , srv1916485313:84 , srv401882966:84 , srv667381576:84 , srv1034226518:85 , srv1039266885:85 , srv1069773820:85 , srv1206844611:85 , srv1383916290:85 , srv1386728933:85 , srv1409806377:85 , srv1411605209:85 , srv1429471019:85 , srv1605371865:85 , srv1615693801:85 , srv171382807:85 , srv1725417758:85 , srv1960010711:85 , srv1973662389:85 , srv393279496:85 , srv726065041:85 , srv812841426:85 , srv1410261826:86 , srv145229301:86 , srv1509559165:86 , srv1650470769:86 , srv1760818363:86 , srv1791605415:86 , srv1821429752:86 , srv304460622:86 , srv374904780:86 , srv486440582:86 , srv532447706:86 , srv694342788:86 , srv911448119:86 , srv1154154547:87 , srv1355822952:87 , srv137177329:87 , srv1407743990:87 , srv1417038201:87 , srv1567449060:87 , srv1805117166:87 , srv1963688432:87 , srv346085983:87 , srv360636389:87 , srv384215703:87 , srv419049197:87 , srv568165290:87 , srv646974177:87 , srv699832196:87 , srv747159586:87 , srv1141373266:88 , srv1373617352:88 , srv1653374971:88 , srv1788974272:88 , srv2039541758:88 , srv2110072049:88 , srv230140836:88 , srv243443735:88 , srv54488016:88 , srv827317314:88 , srv98004355:88 , srv1692003641:89 , srv1872833825:89 , srv422444611:89 , srv478038710:89 , srv944719594:89 , srv948344341:89 , srv1726518293:90 , srv2008079088:90 , srv238745902:90 , srv1195637595:91 , srv1405758175:91 , srv1434621581:91 , srv1137559323:92 , srv127162015:516 } 2024-02-11 05:53:14,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 05:53:14,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2110072049=61, srv1791605415=49, srv1405758175=23, srv667381576=84, srv948344341=98, srv1154154547=8, srv98004355=99, srv1144844548=7, srv171382807=43, srv1821429752=52, srv1116486723=4, srv1069773820=3, srv1434621581=30, srv1760818363=46, srv1963688432=57, srv238745902=64, srv1783985405=47, srv812841426=92, srv694342788=86, srv911448119=95, srv1726518293=45, srv486440582=79, srv346085983=69, srv776132788=91, srv1417038201=28, srv155773995=34, srv1650470769=40, srv1692003641=42, srv1034226518=0, srv568165290=82, srv1195637595=10, srv159956322=37, srv1332241794=15, srv944719594=97, srv1137559323=5, srv393279496=74, srv1805117166=51, srv1043126507=2, srv1491816555=32, srv137177329=18, srv1788974272=48, srv2008079088=59, srv1039266885=1, srv1154554301=9, srv726065041=88, srv1385560946=21, srv1653374971=41, srv883293631=94, srv304460622=67, srv1429471019=29, srv1799191199=50, srv1960010711=56, srv243443735=65, srv127162015=12, srv532447706=80, srv1509559165=33, srv343411892=68, srv145229301=31, srv699832196=87, srv1567449060=35, srv747159586=89, srv1141373266=6, srv1373617352=19, srv1274807635=13, srv1407743990=24, srv1725417758=44, srv230140836=62, srv384215703=73, srv1281395006=14, srv1383916290=20, srv54488016=81, srv827317314=93, srv376976694=72, srv1605371865=38, srv646974177=83, srv66935700=85, srv360636389=70, srv1386728933=22, srv1569704449=36, srv1916485313=55, srv28694002=66, srv374904780=71, srv401882966=75, srv2039541758=60, srv1878116048=54, srv1355822952=17, srv1410261826=26, srv1973662389=58, srv419049197=76, srv938782263=96, srv751092234=90, srv1206844611=11, srv1409806377=25, srv1872833825=53, srv478038710=78, srv1339063797=16, srv1615693801=39, srv422444611=77, srv232642562=63, srv1411605209=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 05:53:14,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 05:53:14,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 05:53:14,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 05:53:14,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 05:53:14,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 05:53:14,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 05:53:14,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 05:53:14,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 05:53:14,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 05:53:14,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 05:53:14,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 05:53:14,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 05:53:14,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 05:53:14,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 05:53:14,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 05:53:14,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 05:53:14,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 05:53:14,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 05:53:14,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 05:53:14,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 05:53:14,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 05:53:14,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 05:53:14,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 05:53:14,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 05:53:14,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 05:53:14,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 05:53:14,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 05:53:14,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 05:53:14,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 05:53:14,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 05:53:14,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 05:53:14,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 05:53:14,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 05:53:14,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 05:53:14,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 05:53:14,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 05:53:14,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 05:53:14,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 05:53:14,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 05:53:14,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 05:53:14,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 05:53:14,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 05:53:14,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 05:53:14,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 05:53:14,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 05:53:14,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 05:53:14,287 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023740501740078745); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03679042475339682, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023740501740078745); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:53:14,287 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 05:53:14,288 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01647742599704986, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023740501740078745); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03679042475339682, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023740501740078745); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04658333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 05:53:14,639 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 485 ms to try 5000 different iterations. Found a solution that moves 1033 regions; Going from a computed imbalance of 0.01647742599704986 to a new imbalance of 8.768957987987354E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007461084974795713); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01188232614108062); MoveCostFunction : (multiplier=7.0, imbalance=0.11477777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007461084974795713); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:53:14,641 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1383916290:80 , srv238745902:82 , srv532447706:82 , srv1355822952:83 , srv1692003641:83 , srv1760818363:83 , srv1963688432:83 , srv243443735:83 , srv360636389:83 , srv419049197:83 , srv54488016:83 , srv1069773820:84 , srv1116486723:84 , srv1141373266:84 , srv1195637595:84 , srv1509559165:84 , srv1726518293:84 , srv1788974272:84 , srv1791605415:84 , srv230140836:84 , srv393279496:84 , srv422444611:84 , srv568165290:84 , srv726065041:84 , srv827317314:84 , srv1034226518:85 , srv1137559323:85 , srv1154154547:85 , srv1206844611:85 , srv137177329:85 , srv1385560946:85 , srv1407743990:85 , srv1417038201:85 , srv1429471019:85 , srv1434621581:85 , srv145229301:85 , srv1567449060:85 , srv1569704449:85 , srv1605371865:85 , srv1653374971:85 , srv1805117166:85 , srv2039541758:85 , srv28694002:85 , srv346085983:85 , srv401882966:85 , srv478038710:85 , srv883293631:85 , srv944719594:85 , srv1039266885:86 , srv1144844548:86 , srv1339063797:86 , srv1405758175:86 , srv1409806377:86 , srv1410261826:86 , srv1411605209:86 , srv1799191199:86 , srv1872833825:86 , srv1916485313:86 , srv1960010711:86 , srv1973662389:86 , srv2008079088:86 , srv2110072049:86 , srv304460622:86 , srv374904780:86 , srv376976694:86 , srv384215703:86 , srv486440582:86 , srv646974177:86 , srv66935700:86 , srv694342788:86 , srv747159586:86 , srv1043126507:87 , srv1154554301:87 , srv1274807635:87 , srv1332241794:87 , srv1373617352:87 , srv1491816555:87 , srv155773995:87 , srv159956322:87 , srv1650470769:87 , srv171382807:87 , srv1725417758:87 , srv1783985405:87 , srv1878116048:87 , srv232642562:87 , srv343411892:87 , srv667381576:87 , srv699832196:87 , srv751092234:87 , srv812841426:87 , srv911448119:87 , srv948344341:87 , srv98004355:87 , srv1281395006:88 , srv1386728933:88 , srv1821429752:88 , srv776132788:88 , srv1615693801:89 , srv938782263:89 , srv127162015:539 } 2024-02-11 05:53:14,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 05:53:14,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2110072049=61, srv1791605415=49, srv1405758175=23, srv667381576=84, srv948344341=98, srv1154154547=8, srv98004355=99, srv1144844548=7, srv171382807=43, srv1821429752=52, srv1116486723=4, srv1069773820=3, srv1434621581=30, srv1760818363=46, srv1963688432=57, srv238745902=64, srv1783985405=47, srv812841426=92, srv694342788=86, srv911448119=95, srv1726518293=45, srv486440582=79, srv346085983=69, srv776132788=91, srv1417038201=28, srv155773995=34, srv1650470769=40, srv1692003641=42, srv1034226518=0, srv568165290=82, srv1195637595=10, srv159956322=37, srv1332241794=15, srv944719594=97, srv1137559323=5, srv393279496=74, srv1805117166=51, srv1043126507=2, srv1491816555=32, srv137177329=18, srv1788974272=48, srv2008079088=59, srv1039266885=1, srv1154554301=9, srv726065041=88, srv1385560946=21, srv1653374971=41, srv883293631=94, srv304460622=67, srv1429471019=29, srv1799191199=50, srv1960010711=56, srv243443735=65, srv127162015=12, srv532447706=80, srv1509559165=33, srv343411892=68, srv145229301=31, srv699832196=87, srv1567449060=35, srv747159586=89, srv1141373266=6, srv1373617352=19, srv1274807635=13, srv1407743990=24, srv1725417758=44, srv230140836=62, srv384215703=73, srv1281395006=14, srv1383916290=20, srv54488016=81, srv827317314=93, srv376976694=72, srv1605371865=38, srv646974177=83, srv66935700=85, srv360636389=70, srv1386728933=22, srv1569704449=36, srv1916485313=55, srv28694002=66, srv374904780=71, srv401882966=75, srv2039541758=60, srv1878116048=54, srv1355822952=17, srv1410261826=26, srv1973662389=58, srv419049197=76, srv938782263=96, srv751092234=90, srv1206844611=11, srv1409806377=25, srv1872833825=53, srv478038710=78, srv1339063797=16, srv1615693801=39, srv422444611=77, srv232642562=63, srv1411605209=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 05:53:14,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 05:53:14,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 05:53:14,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 05:53:14,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 05:53:14,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 05:53:14,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 05:53:14,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 05:53:14,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 05:53:14,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 05:53:14,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 05:53:14,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 05:53:14,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 05:53:14,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 05:53:14,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 05:53:14,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 05:53:14,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 05:53:14,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 05:53:14,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 05:53:14,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 05:53:14,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 05:53:14,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 05:53:14,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 05:53:14,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 05:53:14,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 05:53:14,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 05:53:14,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 05:53:14,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 05:53:14,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 05:53:14,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 05:53:14,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 05:53:14,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 05:53:14,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 05:53:14,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 05:53:14,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 05:53:14,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 05:53:14,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 05:53:14,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 05:53:14,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 05:53:14,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 05:53:14,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 05:53:14,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 05:53:14,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 05:53:14,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 05:53:14,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 05:53:14,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 05:53:14,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 05:53:14,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 05:53:14,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 05:53:14,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 05:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 05:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 05:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 05:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 05:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 05:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 05:53:14,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 05:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 05:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 05:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 05:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 05:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 05:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 05:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 05:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 05:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 05:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 05:53:14,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 05:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 05:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 05:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 05:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 05:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 05:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 05:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 05:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 05:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 05:53:14,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 05:53:14,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 05:53:14,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 05:53:14,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 05:53:14,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 05:53:14,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 05:53:14,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 05:53:14,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 05:53:14,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 05:53:14,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 05:53:14,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 05:53:14,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 05:53:14,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 05:53:14,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 05:53:14,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 05:53:14,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 05:53:14,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 05:53:14,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 05:53:14,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 05:53:14,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 05:53:14,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 05:53:14,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 05:53:14,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 05:53:14,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 05:53:14,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 05:53:14,835 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007461084974795713); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01188232614108062); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007461084974795713); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:53:14,835 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 05:53:14,835 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.696625782000115E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007461084974795713); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01188232614108062); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007461084974795713); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 05:53:15,239 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 535 ms to try 5000 different iterations. Found a solution that moves 324 regions; Going from a computed imbalance of 8.696625782000115E-4 to a new imbalance of 6.753549474602996E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.036, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.833333333333334E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:53:15,240 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv948344341:82 , srv1411605209:83 , srv1615693801:83 , srv1872833825:83 , srv486440582:83 , srv827317314:83 , srv1034226518:84 , srv1116486723:84 , srv1154154547:84 , srv1206844611:84 , srv1339063797:84 , srv1409806377:84 , srv1410261826:84 , srv1417038201:84 , srv145229301:84 , srv1491816555:84 , srv1509559165:84 , srv155773995:84 , srv1567449060:84 , srv1569704449:84 , srv159956322:84 , srv1650470769:84 , srv1653374971:84 , srv1791605415:84 , srv1821429752:84 , srv2110072049:84 , srv230140836:84 , srv232642562:84 , srv478038710:84 , srv568165290:84 , srv646974177:84 , srv694342788:84 , srv747159586:84 , srv776132788:84 , srv883293631:84 , srv911448119:84 , srv938782263:84 , srv1039266885:85 , srv1043126507:85 , srv1137559323:85 , srv1144844548:85 , srv1195637595:85 , srv1274807635:85 , srv1281395006:85 , srv1332241794:85 , srv1355822952:85 , srv137177329:85 , srv1373617352:85 , srv1383916290:85 , srv1385560946:85 , srv1386728933:85 , srv1405758175:85 , srv1407743990:85 , srv1434621581:85 , srv1605371865:85 , srv1692003641:85 , srv171382807:85 , srv1725417758:85 , srv1726518293:85 , srv1760818363:85 , srv1783985405:85 , srv1788974272:85 , srv1805117166:85 , srv1878116048:85 , srv1916485313:85 , srv1960010711:85 , srv1973662389:85 , srv2008079088:85 , srv2039541758:85 , srv238745902:85 , srv243443735:85 , srv28694002:85 , srv304460622:85 , srv343411892:85 , srv360636389:85 , srv376976694:85 , srv401882966:85 , srv419049197:85 , srv422444611:85 , srv54488016:85 , srv667381576:85 , srv699832196:85 , srv726065041:85 , srv751092234:85 , srv812841426:85 , srv944719594:85 , srv98004355:85 , srv1069773820:86 , srv1141373266:86 , srv1154554301:86 , srv1429471019:86 , srv1799191199:86 , srv1963688432:86 , srv346085983:86 , srv374904780:86 , srv384215703:86 , srv393279496:86 , srv532447706:86 , srv66935700:86 , srv127162015:617 } 2024-02-11 05:53:15,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 05:53:15,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2110072049=61, srv1791605415=49, srv1405758175=23, srv667381576=84, srv948344341=98, srv1154154547=8, srv98004355=99, srv1144844548=7, srv171382807=43, srv1821429752=52, srv1116486723=4, srv1069773820=3, srv1434621581=30, srv1760818363=46, srv1963688432=57, srv238745902=64, srv1783985405=47, srv812841426=92, srv694342788=86, srv911448119=95, srv1726518293=45, srv486440582=79, srv346085983=69, srv776132788=91, srv1417038201=28, srv155773995=34, srv1650470769=40, srv1692003641=42, srv1034226518=0, srv568165290=82, srv1195637595=10, srv159956322=37, srv1332241794=15, srv944719594=97, srv1137559323=5, srv393279496=74, srv1805117166=51, srv1043126507=2, srv1491816555=32, srv137177329=18, srv1788974272=48, srv2008079088=59, srv1039266885=1, srv1154554301=9, srv726065041=88, srv1385560946=21, srv1653374971=41, srv883293631=94, srv304460622=67, srv1429471019=29, srv1799191199=50, srv1960010711=56, srv243443735=65, srv127162015=12, srv532447706=80, srv1509559165=33, srv343411892=68, srv145229301=31, srv699832196=87, srv1567449060=35, srv747159586=89, srv1141373266=6, srv1373617352=19, srv1274807635=13, srv1407743990=24, srv1725417758=44, srv230140836=62, srv384215703=73, srv1281395006=14, srv1383916290=20, srv54488016=81, srv827317314=93, srv376976694=72, srv1605371865=38, srv646974177=83, srv66935700=85, srv360636389=70, srv1386728933=22, srv1569704449=36, srv1916485313=55, srv28694002=66, srv374904780=71, srv401882966=75, srv2039541758=60, srv1878116048=54, srv1355822952=17, srv1410261826=26, srv1973662389=58, srv419049197=76, srv938782263=96, srv751092234=90, srv1206844611=11, srv1409806377=25, srv1872833825=53, srv478038710=78, srv1339063797=16, srv1615693801=39, srv422444611=77, srv232642562=63, srv1411605209=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 05:53:15,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 05:53:15,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 05:53:15,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 05:53:15,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 05:53:15,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 05:53:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 05:53:15,355 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.833333333333334E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:53:15,356 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 05:53:15,356 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.526679825620759E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.833333333333334E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 05:53:15,599 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 332 ms to try 5000 different iterations. Found a solution that moves 52 regions; Going from a computed imbalance of 6.526679825620759E-5 to a new imbalance of 7.437351078646619E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0057777777777777775); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:53:15,599 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1274807635:83 , srv1332241794:83 , srv1373617352:83 , srv1615693801:83 , srv2008079088:83 , srv374904780:83 , srv384215703:83 , srv393279496:83 , srv66935700:83 , srv699832196:83 , srv98004355:83 , srv1039266885:84 , srv1043126507:84 , srv1069773820:84 , srv1116486723:84 , srv1137559323:84 , srv1141373266:84 , srv1144844548:84 , srv1154554301:84 , srv1195637595:84 , srv1206844611:84 , srv1281395006:84 , srv1339063797:84 , srv1355822952:84 , srv137177329:84 , srv1385560946:84 , srv1386728933:84 , srv1405758175:84 , srv1407743990:84 , srv1409806377:84 , srv1411605209:84 , srv1417038201:84 , srv1429471019:84 , srv1434621581:84 , srv145229301:84 , srv1491816555:84 , srv1509559165:84 , srv1567449060:84 , srv1569704449:84 , srv159956322:84 , srv1605371865:84 , srv1653374971:84 , srv1692003641:84 , srv1725417758:84 , srv1726518293:84 , srv1760818363:84 , srv1783985405:84 , srv1788974272:84 , srv1791605415:84 , srv1799191199:84 , srv1805117166:84 , srv1821429752:84 , srv1872833825:84 , srv1878116048:84 , srv1916485313:84 , srv1960010711:84 , srv1963688432:84 , srv1973662389:84 , srv2039541758:84 , srv2110072049:84 , srv230140836:84 , srv232642562:84 , srv238745902:84 , srv28694002:84 , srv304460622:84 , srv343411892:84 , srv346085983:84 , srv360636389:84 , srv376976694:84 , srv401882966:84 , srv419049197:84 , srv422444611:84 , srv478038710:84 , srv532447706:84 , srv54488016:84 , srv568165290:84 , srv646974177:84 , srv667381576:84 , srv694342788:84 , srv726065041:84 , srv747159586:84 , srv776132788:84 , srv812841426:84 , srv827317314:84 , srv883293631:84 , srv911448119:84 , srv938782263:84 , srv944719594:84 , srv948344341:84 , srv1034226518:85 , srv1154154547:85 , srv1383916290:85 , srv1410261826:85 , srv155773995:85 , srv1650470769:85 , srv171382807:85 , srv243443735:85 , srv486440582:85 , srv751092234:85 , srv127162015:685 } 2024-02-11 05:53:15,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 05:53:15,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2110072049=61, srv1791605415=49, srv1405758175=23, srv667381576=84, srv948344341=98, srv1154154547=8, srv98004355=99, srv1144844548=7, srv171382807=43, srv1821429752=52, srv1116486723=4, srv1069773820=3, srv1434621581=30, srv1760818363=46, srv1963688432=57, srv238745902=64, srv1783985405=47, srv812841426=92, srv694342788=86, srv911448119=95, srv1726518293=45, srv486440582=79, srv346085983=69, srv776132788=91, srv1417038201=28, srv155773995=34, srv1650470769=40, srv1692003641=42, srv1034226518=0, srv568165290=82, srv1195637595=10, srv159956322=37, srv1332241794=15, srv944719594=97, srv1137559323=5, srv393279496=74, srv1805117166=51, srv1043126507=2, srv1491816555=32, srv137177329=18, srv1788974272=48, srv2008079088=59, srv1039266885=1, srv1154554301=9, srv726065041=88, srv1385560946=21, srv1653374971=41, srv883293631=94, srv304460622=67, srv1429471019=29, srv1799191199=50, srv1960010711=56, srv243443735=65, srv127162015=12, srv532447706=80, srv1509559165=33, srv343411892=68, srv145229301=31, srv699832196=87, srv1567449060=35, srv747159586=89, srv1141373266=6, srv1373617352=19, srv1274807635=13, srv1407743990=24, srv1725417758=44, srv230140836=62, srv384215703=73, srv1281395006=14, srv1383916290=20, srv54488016=81, srv827317314=93, srv376976694=72, srv1605371865=38, srv646974177=83, srv66935700=85, srv360636389=70, srv1386728933=22, srv1569704449=36, srv1916485313=55, srv28694002=66, srv374904780=71, srv401882966=75, srv2039541758=60, srv1878116048=54, srv1355822952=17, srv1410261826=26, srv1973662389=58, srv419049197=76, srv938782263=96, srv751092234=90, srv1206844611=11, srv1409806377=25, srv1872833825=53, srv478038710=78, srv1339063797=16, srv1615693801=39, srv422444611=77, srv232642562=63, srv1411605209=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 05:53:15,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 05:53:15,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 05:53:15,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 05:53:15,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 05:53:15,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 05:53:15,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 05:53:15,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 05:53:15,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 05:53:15,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 05:53:15,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 05:53:15,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 05:53:15,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 05:53:15,718 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:53:15,718 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1274807635:83 , srv1332241794:83 , srv1373617352:83 , srv1615693801:83 , srv2008079088:83 , srv374904780:83 , srv384215703:83 , srv393279496:83 , srv66935700:83 , srv699832196:83 , srv98004355:83 , srv1039266885:84 , srv1043126507:84 , srv1069773820:84 , srv1116486723:84 , srv1137559323:84 , srv1141373266:84 , srv1144844548:84 , srv1154554301:84 , srv1195637595:84 , srv1206844611:84 , srv1281395006:84 , srv1339063797:84 , srv1355822952:84 , srv137177329:84 , srv1385560946:84 , srv1386728933:84 , srv1405758175:84 , srv1407743990:84 , srv1409806377:84 , srv1411605209:84 , srv1417038201:84 , srv1429471019:84 , srv1434621581:84 , srv145229301:84 , srv1491816555:84 , srv1509559165:84 , srv1567449060:84 , srv1569704449:84 , srv159956322:84 , srv1605371865:84 , srv1653374971:84 , srv1692003641:84 , srv1725417758:84 , srv1726518293:84 , srv1760818363:84 , srv1783985405:84 , srv1788974272:84 , srv1791605415:84 , srv1799191199:84 , srv1805117166:84 , srv1821429752:84 , srv1872833825:84 , srv1878116048:84 , srv1916485313:84 , srv1960010711:84 , srv1963688432:84 , srv1973662389:84 , srv2039541758:84 , srv2110072049:84 , srv230140836:84 , srv232642562:84 , srv238745902:84 , srv28694002:84 , srv304460622:84 , srv343411892:84 , srv346085983:84 , srv360636389:84 , srv376976694:84 , srv401882966:84 , srv419049197:84 , srv422444611:84 , srv478038710:84 , srv532447706:84 , srv54488016:84 , srv568165290:84 , srv646974177:84 , srv667381576:84 , srv694342788:84 , srv726065041:84 , srv747159586:84 , srv776132788:84 , srv812841426:84 , srv827317314:84 , srv883293631:84 , srv911448119:84 , srv938782263:84 , srv944719594:84 , srv948344341:84 , srv1034226518:85 , srv1154154547:85 , srv1383916290:85 , srv1410261826:85 , srv155773995:85 , srv1650470769:85 , srv171382807:85 , srv243443735:85 , srv486440582:85 , srv751092234:85 , srv127162015:685 } 2024-02-11 05:53:15,740 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=262 (was 224) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=2400 (was 3159) 2024-02-11 05:53:15,747 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=262, ProcessCount=170, AvailableMemoryMB=2400 2024-02-11 05:53:15,747 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-11 05:53:15,747 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-02-11 05:53:15,755 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1116514312:84 , srv1141675658:84 , srv1254533773:84 , srv1424636320:84 , srv1601267165:84 , srv1649955530:84 , srv1706306109:84 , srv1737502028:84 , srv1787787290:84 , srv1788319981:84 , srv1862209693:84 , srv1932797889:84 , srv2020040035:84 , srv2036770107:84 , srv2045595985:84 , srv2063128095:84 , srv260984976:84 , srv290868610:84 , srv332958545:84 , srv413862759:84 , srv418216370:84 , srv559950118:84 , srv616810550:84 , srv658751994:84 , srv688524110:84 , srv82551473:84 , srv836101653:84 , srv882145303:84 , srv888281090:84 , srv266501532:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-02-11 05:53:15,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 05:53:15,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv290868610=18, srv1737502028=7, srv1788319981=9, srv2063128095=15, srv1141675658=1, srv1787787290=8, srv2020040035=12, srv888281090=29, srv688524110=25, srv1649955530=5, srv1116514312=0, srv559950118=22, srv2045595985=14, srv1932797889=11, srv413862759=20, srv836101653=27, srv882145303=28, srv1254533773=2, srv266501532=17, srv260984976=16, srv418216370=21, srv658751994=24, srv332958545=19, srv616810550=23, srv2036770107=13, srv1601267165=4, srv1706306109=6, srv1424636320=3, srv82551473=26, srv1862209693=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 05:53:15,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 05:53:15,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 05:53:15,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 05:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 05:53:15,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 05:53:15,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 05:53:15,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-11 05:53:15,780 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1425857777953268, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:53:15,780 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912642326394259, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1425857777953268, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-02-11 05:55:15,769 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6334097 different iterations. Found a solution that moves 1859 regions; Going from a computed imbalance of 0.9912642326394259 to a new imbalance of 4.338998739279625E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6885185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:55:15,776 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1116514312:90 , srv1141675658:90 , srv1254533773:90 , srv1424636320:90 , srv1601267165:90 , srv1649955530:90 , srv1706306109:90 , srv1737502028:90 , srv1787787290:90 , srv1788319981:90 , srv1862209693:90 , srv1932797889:90 , srv2020040035:90 , srv2036770107:90 , srv2045595985:90 , srv2063128095:90 , srv260984976:90 , srv266501532:90 , srv290868610:90 , srv332958545:90 , srv413862759:90 , srv418216370:90 , srv559950118:90 , srv616810550:90 , srv658751994:90 , srv688524110:90 , srv82551473:90 , srv836101653:90 , srv882145303:90 , srv888281090:90 } 2024-02-11 05:55:15,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 05:55:15,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv290868610=18, srv1737502028=7, srv1788319981=9, srv2063128095=15, srv1141675658=1, srv1787787290=8, srv2020040035=12, srv888281090=29, srv688524110=25, srv1649955530=5, srv1116514312=0, srv559950118=22, srv2045595985=14, srv1932797889=11, srv413862759=20, srv836101653=27, srv882145303=28, srv1254533773=2, srv266501532=17, srv260984976=16, srv418216370=21, srv658751994=24, srv332958545=19, srv616810550=23, srv2036770107=13, srv1601267165=4, srv1706306109=6, srv1424636320=3, srv82551473=26, srv1862209693=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 05:55:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 05:55:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 05:55:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 05:55:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 05:55:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 05:55:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 05:55:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 05:55:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 05:55:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 05:55:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 05:55:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-11 05:55:15,798 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 05:55:15,798 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1116514312:90 , srv1141675658:90 , srv1254533773:90 , srv1424636320:90 , srv1601267165:90 , srv1649955530:90 , srv1706306109:90 , srv1737502028:90 , srv1787787290:90 , srv1788319981:90 , srv1862209693:90 , srv1932797889:90 , srv2020040035:90 , srv2036770107:90 , srv2045595985:90 , srv2063128095:90 , srv260984976:90 , srv266501532:90 , srv290868610:90 , srv332958545:90 , srv413862759:90 , srv418216370:90 , srv559950118:90 , srv616810550:90 , srv658751994:90 , srv688524110:90 , srv82551473:90 , srv836101653:90 , srv882145303:90 , srv888281090:90 } 2024-02-11 05:55:15,810 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=188 (was 262), ProcessCount=168 (was 170), AvailableMemoryMB=1982 (was 2400)