2023-12-19 13:53:15,110 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-19 13:53:15,434 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-19 13:53:15,556 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-19 13:53:15,625 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-19 13:53:15,632 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-19 13:53:15,645 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=276, ProcessCount=170, AvailableMemoryMB=3717 2023-12-19 13:53:15,648 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-19 13:53:15,648 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-19 13:53:15,787 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1019776688:84 , srv1020466726:84 , srv1030923475:84 , srv1071065244:84 , srv1100818659:84 , srv1102697018:84 , srv1114253798:84 , srv1130862155:84 , srv1135440549:84 , srv1150332352:84 , srv117156483:84 , srv1257544769:84 , srv1258748155:84 , srv1258905363:84 , srv1306599683:84 , srv1312385521:84 , srv1360245195:84 , srv1363664119:84 , srv1364203954:84 , srv140998937:84 , srv1438651312:84 , srv1448235260:84 , srv1467967424:84 , srv1471578074:84 , srv1472867013:84 , srv1480802206:84 , srv1486844457:84 , srv1544750042:84 , srv1547258889:84 , srv1605533195:84 , srv1619763671:84 , srv1624626063:84 , srv1676370661:84 , srv1685147705:84 , srv1736494758:84 , srv1739762035:84 , srv1743935831:84 , srv1755037149:84 , srv1794719650:84 , srv182277395:84 , srv1831215936:84 , srv1862675081:84 , srv1904315277:84 , srv1935637408:84 , srv1937248082:84 , srv1938550787:84 , srv1945580515:84 , srv1995346797:84 , srv2010283524:84 , srv2010838139:84 , srv2014843238:84 , srv2061582561:84 , srv2072570318:84 , srv2073414701:84 , srv2082024426:84 , srv2119650479:84 , srv2122898565:84 , srv2133297157:84 , srv234003506:84 , srv238220953:84 , srv321704973:84 , srv32448664:84 , srv342628156:84 , srv429027790:84 , srv44441552:84 , srv50187028:84 , srv5063628:84 , srv516547335:84 , srv540989264:84 , srv555310386:84 , srv560786850:84 , srv568762576:84 , srv609235308:84 , srv628884539:84 , srv722106953:84 , srv729898444:84 , srv737583930:84 , srv7413056:84 , srv743240987:84 , srv748430389:84 , srv761538373:84 , srv80825674:84 , srv819853020:84 , srv823453102:84 , srv848342604:84 , srv850753440:84 , srv862108761:84 , srv869227296:84 , srv883319831:84 , srv885816629:84 , srv888661502:84 , srv914424623:84 , srv927426340:84 , srv9364796:84 , srv952378590:84 , srv967567616:84 , srv979191340:84 , srv98286284:84 , srv992194527:84 , srv801627502:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-19 13:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-19 13:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv748430389=79, srv1862675081=41, srv182277395=39, srv729898444=75, srv1743935831=36, srv1448235260=21, srv722106953=74, srv1130862155=7, srv2122898565=56, srv1071065244=3, srv1605533195=29, srv1312385521=15, srv98286284=98, srv1794719650=38, srv1114253798=6, srv2082024426=54, srv862108761=87, srv1364203954=18, srv1685147705=33, srv848342604=85, srv1547258889=28, srv2073414701=53, srv819853020=83, srv1619763671=30, srv50187028=65, srv888661502=91, srv761538373=80, srv952378590=95, srv1624626063=31, srv32448664=61, srv568762576=71, srv321704973=60, srv1257544769=11, srv1676370661=32, srv1486844457=26, srv238220953=59, srv516547335=67, srv885816629=90, srv1755037149=37, srv1020466726=1, srv1935637408=43, srv1736494758=34, srv5063628=66, srv2072570318=52, srv1471578074=23, srv1831215936=40, srv1467967424=22, srv743240987=78, srv869227296=88, srv429027790=63, srv1904315277=42, srv1019776688=0, srv2133297157=57, srv234003506=58, srv992194527=99, srv1360245195=16, srv1995346797=47, srv823453102=84, srv628884539=73, srv2119650479=55, srv44441552=64, srv883319831=89, srv1945580515=46, srv1480802206=25, srv555310386=69, srv2010838139=49, srv2014843238=50, srv1938550787=45, srv1438651312=20, srv1100818659=4, srv1135440549=8, srv967567616=96, srv1937248082=44, srv801627502=81, srv1102697018=5, srv737583930=76, srv560786850=70, srv1544750042=27, srv1306599683=14, srv540989264=68, srv117156483=10, srv140998937=19, srv1363664119=17, srv927426340=93, srv1030923475=2, srv7413056=77, srv1258905363=13, srv342628156=62, srv914424623=92, srv850753440=86, srv1258748155=12, srv1739762035=35, srv2010283524=48, srv2061582561=51, srv609235308=72, srv1150332352=9, srv80825674=82, srv9364796=94, srv979191340=97, srv1472867013=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-19 13:53:15,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-19 13:53:15,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-19 13:53:15,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-19 13:53:15,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-19 13:53:15,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-19 13:53:15,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-19 13:53:15,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-19 13:53:15,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-19 13:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-19 13:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-19 13:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-19 13:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-19 13:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-19 13:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-19 13:53:15,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-19 13:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-19 13:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-19 13:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-19 13:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-19 13:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-19 13:53:15,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-19 13:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-19 13:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-19 13:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-19 13:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-19 13:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-19 13:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-19 13:53:15,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-19 13:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-19 13:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-19 13:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-19 13:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-19 13:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-19 13:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-19 13:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-19 13:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-19 13:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-19 13:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-19 13:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-19 13:53:15,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-19 13:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-19 13:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-19 13:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-19 13:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-19 13:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-19 13:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-19 13:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-19 13:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-19 13:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-19 13:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-19 13:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-19 13:53:15,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-19 13:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-19 13:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-19 13:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-19 13:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-19 13:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-19 13:53:15,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-19 13:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-19 13:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-19 13:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-19 13:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-19 13:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-19 13:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-19 13:53:15,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-19 13:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-19 13:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-19 13:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-19 13:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-19 13:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-19 13:53:15,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-19 13:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-19 13:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-19 13:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-19 13:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-19 13:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-19 13:53:15,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-19 13:53:15,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-19 13:53:15,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-19 13:53:15,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-19 13:53:15,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-19 13:53:15,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-19 13:53:15,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-19 13:53:15,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-19 13:53:15,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-19 13:53:15,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-19 13:53:15,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-19 13:53:15,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-19 13:53:15,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-19 13:53:15,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-19 13:53:15,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-19 13:53:15,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-19 13:53:15,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-19 13:53:15,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-19 13:53:15,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-19 13:53:15,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-19 13:53:15,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-19 13:53:15,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-19 13:53:15,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-19 13:53:15,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-19 13:53:15,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-19 13:53:16,136 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:53:16,137 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-19 13:53:16,137 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-19 13:53:16,522 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 555 ms to try 5000 different iterations. Found a solution that moves 4658 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18470202192225224. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03995164804259477, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055632270266002044, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5175555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03995164804259477, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17983333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.248, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:53:16,545 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv761538373:81 , srv1102697018:82 , srv1150332352:82 , srv1486844457:82 , srv5063628:82 , srv869227296:82 , srv885816629:82 , srv914424623:82 , srv1364203954:83 , srv1071065244:84 , srv1258905363:84 , srv1312385521:84 , srv1363664119:84 , srv140998937:84 , srv1935637408:84 , srv234003506:84 , srv540989264:84 , srv628884539:84 , srv729898444:84 , srv748430389:84 , srv819853020:84 , srv823453102:84 , srv1019776688:85 , srv1130862155:85 , srv1257544769:85 , srv1360245195:85 , srv1438651312:85 , srv1472867013:85 , srv1544750042:85 , srv1755037149:85 , srv1862675081:85 , srv1938550787:85 , srv1995346797:85 , srv2072570318:85 , srv238220953:85 , srv342628156:85 , srv7413056:85 , srv80825674:85 , srv888661502:85 , srv992194527:85 , srv117156483:86 , srv1258748155:86 , srv1739762035:86 , srv1904315277:86 , srv1945580515:86 , srv2010838139:86 , srv2061582561:86 , srv2133297157:86 , srv560786850:86 , srv737583930:86 , srv743240987:86 , srv850753440:86 , srv9364796:86 , srv967567616:86 , srv979191340:86 , srv1100818659:87 , srv1114253798:87 , srv1135440549:87 , srv1306599683:87 , srv1467967424:87 , srv1624626063:87 , srv182277395:87 , srv1831215936:87 , srv2014843238:87 , srv321704973:87 , srv429027790:87 , srv50187028:87 , srv609235308:87 , srv952378590:87 , srv1547258889:88 , srv1619763671:88 , srv1685147705:88 , srv1743935831:88 , srv1937248082:88 , srv2010283524:88 , srv32448664:88 , srv44441552:88 , srv516547335:88 , srv848342604:88 , srv98286284:88 , srv1030923475:89 , srv1471578074:89 , srv1480802206:89 , srv1794719650:89 , srv2073414701:89 , srv2082024426:89 , srv2122898565:89 , srv862108761:89 , srv1020466726:90 , srv1605533195:90 , srv1676370661:90 , srv1736494758:91 , srv555310386:91 , srv568762576:91 , srv883319831:91 , srv927426340:91 , srv1448235260:93 , srv2119650479:94 , srv722106953:95 , srv801627502:445 } 2023-12-19 13:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-19 13:53:16,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv748430389=79, srv1862675081=41, srv182277395=39, srv729898444=75, srv1743935831=36, srv1448235260=21, srv722106953=74, srv1130862155=7, srv2122898565=56, srv1071065244=3, srv1605533195=29, srv1312385521=15, srv98286284=98, srv1794719650=38, srv1114253798=6, srv2082024426=54, srv862108761=87, srv1364203954=18, srv1685147705=33, srv848342604=85, srv1547258889=28, srv2073414701=53, srv819853020=83, srv1619763671=30, srv50187028=65, srv888661502=91, srv761538373=80, srv952378590=95, srv1624626063=31, srv32448664=61, srv568762576=71, srv321704973=60, srv1257544769=11, srv1676370661=32, srv1486844457=26, srv238220953=59, srv516547335=67, srv885816629=90, srv1755037149=37, srv1020466726=1, srv1935637408=43, srv1736494758=34, srv5063628=66, srv2072570318=52, srv1471578074=23, srv1831215936=40, srv1467967424=22, srv743240987=78, srv869227296=88, srv429027790=63, srv1904315277=42, srv1019776688=0, srv2133297157=57, srv234003506=58, srv992194527=99, srv1360245195=16, srv1995346797=47, srv823453102=84, srv628884539=73, srv2119650479=55, srv44441552=64, srv883319831=89, srv1945580515=46, srv1480802206=25, srv555310386=69, srv2010838139=49, srv2014843238=50, srv1938550787=45, srv1438651312=20, srv1100818659=4, srv1135440549=8, srv967567616=96, srv1937248082=44, srv801627502=81, srv1102697018=5, srv737583930=76, srv560786850=70, srv1544750042=27, srv1306599683=14, srv540989264=68, srv117156483=10, srv140998937=19, srv1363664119=17, srv927426340=93, srv1030923475=2, srv7413056=77, srv1258905363=13, srv342628156=62, srv914424623=92, srv850753440=86, srv1258748155=12, srv1739762035=35, srv2010283524=48, srv2061582561=51, srv609235308=72, srv1150332352=9, srv80825674=82, srv9364796=94, srv979191340=97, srv1472867013=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-19 13:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-19 13:53:16,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-19 13:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-19 13:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-19 13:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-19 13:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-19 13:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-19 13:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-19 13:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-19 13:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-19 13:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-19 13:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-19 13:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-19 13:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-19 13:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-19 13:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-19 13:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-19 13:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-19 13:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-19 13:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-19 13:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-19 13:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-19 13:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-19 13:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-19 13:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-19 13:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-19 13:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-19 13:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-19 13:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-19 13:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-19 13:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-19 13:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-19 13:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-19 13:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-19 13:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-19 13:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-19 13:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-19 13:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-19 13:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-19 13:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-19 13:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-19 13:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-19 13:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-19 13:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-19 13:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-19 13:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-19 13:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-19 13:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-19 13:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-19 13:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-19 13:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-19 13:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-19 13:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-19 13:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-19 13:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-19 13:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-19 13:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-19 13:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-19 13:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-19 13:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-19 13:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-19 13:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-19 13:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-19 13:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-19 13:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-19 13:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-19 13:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-19 13:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-19 13:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-19 13:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-19 13:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-19 13:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-19 13:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-19 13:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-19 13:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-19 13:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-19 13:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-19 13:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-19 13:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-19 13:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-19 13:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-19 13:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-19 13:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-19 13:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-19 13:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-19 13:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-19 13:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-19 13:53:16,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-19 13:53:16,719 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03995164804259477, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055632270266002044, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03995164804259477, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17983333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.248, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:53:16,719 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-19 13:53:16,720 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18466940590913625, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03995164804259477, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055632270266002044, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03995164804259477, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17983333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.248, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-19 13:53:16,963 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 357 ms to try 5000 different iterations. Found a solution that moves 2662 regions; Going from a computed imbalance of 0.18466940590913625 to a new imbalance of 0.015249397945153904. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020478135514954695); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03499559055116361, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29577777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020478135514954695); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:53:16,972 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv722106953:77 , srv1448235260:78 , srv2119650479:80 , srv555310386:80 , srv927426340:80 , srv2082024426:81 , srv568762576:81 , srv883319831:81 , srv1676370661:82 , srv1020466726:83 , srv1258748155:83 , srv1480802206:83 , srv1619763671:83 , srv1736494758:83 , srv1743935831:83 , srv1794719650:83 , srv2010283524:83 , srv321704973:83 , srv44441552:83 , srv1135440549:84 , srv1306599683:84 , srv1467967424:84 , srv1547258889:84 , srv1685147705:84 , srv1937248082:84 , srv2010838139:84 , srv2122898565:84 , srv32448664:84 , srv516547335:84 , srv609235308:84 , srv848342604:84 , srv1019776688:85 , srv1114253798:85 , srv1130862155:85 , srv117156483:85 , srv1605533195:85 , srv1831215936:85 , srv1904315277:85 , srv2014843238:85 , srv50187028:85 , srv743240987:85 , srv862108761:85 , srv1030923475:86 , srv1100818659:86 , srv1360245195:86 , srv1438651312:86 , srv1471578074:86 , srv182277395:86 , srv1935637408:86 , srv1938550787:86 , srv1945580515:86 , srv1995346797:86 , srv2072570318:86 , srv2073414701:86 , srv2133297157:86 , srv238220953:86 , srv342628156:86 , srv80825674:86 , srv850753440:86 , srv979191340:86 , srv98286284:86 , srv992194527:86 , srv140998937:87 , srv1624626063:87 , srv1755037149:87 , srv2061582561:87 , srv429027790:87 , srv560786850:87 , srv628884539:87 , srv737583930:87 , srv7413056:87 , srv819853020:87 , srv9364796:87 , srv967567616:87 , srv1257544769:88 , srv1363664119:88 , srv1544750042:88 , srv1739762035:88 , srv234003506:88 , srv729898444:88 , srv748430389:88 , srv888661502:88 , srv952378590:88 , srv1071065244:89 , srv1258905363:89 , srv1472867013:89 , srv1862675081:89 , srv540989264:89 , srv823453102:89 , srv885816629:89 , srv1102697018:90 , srv1312385521:90 , srv869227296:90 , srv914424623:90 , srv1364203954:91 , srv1486844457:91 , srv5063628:91 , srv1150332352:92 , srv761538373:92 , srv801627502:511 } 2023-12-19 13:53:16,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-19 13:53:16,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv748430389=79, srv1862675081=41, srv182277395=39, srv729898444=75, srv1743935831=36, srv1448235260=21, srv722106953=74, srv1130862155=7, srv2122898565=56, srv1071065244=3, srv1605533195=29, srv1312385521=15, srv98286284=98, srv1794719650=38, srv1114253798=6, srv2082024426=54, srv862108761=87, srv1364203954=18, srv1685147705=33, srv848342604=85, srv1547258889=28, srv2073414701=53, srv819853020=83, srv1619763671=30, srv50187028=65, srv888661502=91, srv761538373=80, srv952378590=95, srv1624626063=31, srv32448664=61, srv568762576=71, srv321704973=60, srv1257544769=11, srv1676370661=32, srv1486844457=26, srv238220953=59, srv516547335=67, srv885816629=90, srv1755037149=37, srv1020466726=1, srv1935637408=43, srv1736494758=34, srv5063628=66, srv2072570318=52, srv1471578074=23, srv1831215936=40, srv1467967424=22, srv743240987=78, srv869227296=88, srv429027790=63, srv1904315277=42, srv1019776688=0, srv2133297157=57, srv234003506=58, srv992194527=99, srv1360245195=16, srv1995346797=47, srv823453102=84, srv628884539=73, srv2119650479=55, srv44441552=64, srv883319831=89, srv1945580515=46, srv1480802206=25, srv555310386=69, srv2010838139=49, srv2014843238=50, srv1938550787=45, srv1438651312=20, srv1100818659=4, srv1135440549=8, srv967567616=96, srv1937248082=44, srv801627502=81, srv1102697018=5, srv737583930=76, srv560786850=70, srv1544750042=27, srv1306599683=14, srv540989264=68, srv117156483=10, srv140998937=19, srv1363664119=17, srv927426340=93, srv1030923475=2, srv7413056=77, srv1258905363=13, srv342628156=62, srv914424623=92, srv850753440=86, srv1258748155=12, srv1739762035=35, srv2010283524=48, srv2061582561=51, srv609235308=72, srv1150332352=9, srv80825674=82, srv9364796=94, srv979191340=97, srv1472867013=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-19 13:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-19 13:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-19 13:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-19 13:53:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-19 13:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-19 13:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-19 13:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-19 13:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-19 13:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-19 13:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-19 13:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-19 13:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-19 13:53:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-19 13:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-19 13:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-19 13:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-19 13:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-19 13:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-19 13:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-19 13:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-19 13:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-19 13:53:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-19 13:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-19 13:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-19 13:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-19 13:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-19 13:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-19 13:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-19 13:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-19 13:53:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-19 13:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-19 13:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-19 13:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-19 13:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-19 13:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-19 13:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-19 13:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-19 13:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-19 13:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-19 13:53:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-19 13:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-19 13:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-19 13:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-19 13:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-19 13:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-19 13:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-19 13:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-19 13:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-19 13:53:17,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-19 13:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-19 13:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-19 13:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-19 13:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-19 13:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-19 13:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-19 13:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-19 13:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-19 13:53:17,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-19 13:53:17,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-19 13:53:17,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-19 13:53:17,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-19 13:53:17,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-19 13:53:17,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-19 13:53:17,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-19 13:53:17,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-19 13:53:17,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-19 13:53:17,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-19 13:53:17,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-19 13:53:17,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-19 13:53:17,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-19 13:53:17,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-19 13:53:17,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-19 13:53:17,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-19 13:53:17,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-19 13:53:17,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-19 13:53:17,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-19 13:53:17,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-19 13:53:17,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-19 13:53:17,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-19 13:53:17,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-19 13:53:17,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-19 13:53:17,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-19 13:53:17,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-19 13:53:17,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-19 13:53:17,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-19 13:53:17,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-19 13:53:17,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-19 13:53:17,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-19 13:53:17,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-19 13:53:17,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-19 13:53:17,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-19 13:53:17,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-19 13:53:17,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-19 13:53:17,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-19 13:53:17,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-19 13:53:17,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-19 13:53:17,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-19 13:53:17,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-19 13:53:17,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-19 13:53:17,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-19 13:53:17,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-19 13:53:17,146 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020478135514954695); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03499559055116361, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020478135514954695); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:53:17,146 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-19 13:53:17,146 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01523075822275913, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020478135514954695); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03499559055116361, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020478135514954695); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04633333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-19 13:53:17,376 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 342 ms to try 5000 different iterations. Found a solution that moves 997 regions; Going from a computed imbalance of 0.01523075822275913 to a new imbalance of 0.001036904556617527. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006662924501529447); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010005610098618013); MoveCostFunction : (multiplier=7.0, imbalance=0.11077777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006662924501529447); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:53:17,379 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1471578074:81 , srv1605533195:82 , srv952378590:82 , srv1030923475:83 , srv1071065244:83 , srv1150332352:83 , srv1257544769:83 , srv1312385521:83 , srv1736494758:83 , srv2073414701:83 , srv429027790:83 , srv98286284:83 , srv1100818659:84 , srv1258905363:84 , srv1364203954:84 , srv1472867013:84 , srv1624626063:84 , srv182277395:84 , srv1862675081:84 , srv2119650479:84 , srv560786850:84 , srv737583930:84 , srv761538373:84 , srv862108761:84 , srv9364796:84 , srv967567616:84 , srv1020466726:85 , srv1102697018:85 , srv1363664119:85 , srv1480802206:85 , srv1486844457:85 , srv1544750042:85 , srv1547258889:85 , srv1685147705:85 , srv1739762035:85 , srv1755037149:85 , srv1794719650:85 , srv1945580515:85 , srv2014843238:85 , srv2061582561:85 , srv2122898565:85 , srv234003506:85 , srv238220953:85 , srv50187028:85 , srv5063628:85 , srv516547335:85 , srv540989264:85 , srv568762576:85 , srv609235308:85 , srv729898444:85 , srv7413056:85 , srv748430389:85 , srv819853020:85 , srv823453102:85 , srv848342604:85 , srv869227296:85 , srv888661502:85 , srv979191340:85 , srv1114253798:86 , srv117156483:86 , srv140998937:86 , srv1438651312:86 , srv1467967424:86 , srv1619763671:86 , srv1676370661:86 , srv1831215936:86 , srv1937248082:86 , srv1938550787:86 , srv1995346797:86 , srv2010283524:86 , srv2133297157:86 , srv44441552:86 , srv628884539:86 , srv722106953:86 , srv80825674:86 , srv850753440:86 , srv914424623:86 , srv927426340:86 , srv992194527:86 , srv1019776688:87 , srv1135440549:87 , srv1306599683:87 , srv1360245195:87 , srv1448235260:87 , srv1743935831:87 , srv1904315277:87 , srv1935637408:87 , srv2010838139:87 , srv2072570318:87 , srv321704973:87 , srv32448664:87 , srv342628156:87 , srv555310386:87 , srv743240987:87 , srv883319831:87 , srv885816629:87 , srv1130862155:88 , srv2082024426:88 , srv1258748155:90 , srv801627502:561 } 2023-12-19 13:53:17,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-19 13:53:17,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv748430389=79, srv1862675081=41, srv182277395=39, srv729898444=75, srv1743935831=36, srv1448235260=21, srv722106953=74, srv1130862155=7, srv2122898565=56, srv1071065244=3, srv1605533195=29, srv1312385521=15, srv98286284=98, srv1794719650=38, srv1114253798=6, srv2082024426=54, srv862108761=87, srv1364203954=18, srv1685147705=33, srv848342604=85, srv1547258889=28, srv2073414701=53, srv819853020=83, srv1619763671=30, srv50187028=65, srv888661502=91, srv761538373=80, srv952378590=95, srv1624626063=31, srv32448664=61, srv568762576=71, srv321704973=60, srv1257544769=11, srv1676370661=32, srv1486844457=26, srv238220953=59, srv516547335=67, srv885816629=90, srv1755037149=37, srv1020466726=1, srv1935637408=43, srv1736494758=34, srv5063628=66, srv2072570318=52, srv1471578074=23, srv1831215936=40, srv1467967424=22, srv743240987=78, srv869227296=88, srv429027790=63, srv1904315277=42, srv1019776688=0, srv2133297157=57, srv234003506=58, srv992194527=99, srv1360245195=16, srv1995346797=47, srv823453102=84, srv628884539=73, srv2119650479=55, srv44441552=64, srv883319831=89, srv1945580515=46, srv1480802206=25, srv555310386=69, srv2010838139=49, srv2014843238=50, srv1938550787=45, srv1438651312=20, srv1100818659=4, srv1135440549=8, srv967567616=96, srv1937248082=44, srv801627502=81, srv1102697018=5, srv737583930=76, srv560786850=70, srv1544750042=27, srv1306599683=14, srv540989264=68, srv117156483=10, srv140998937=19, srv1363664119=17, srv927426340=93, srv1030923475=2, srv7413056=77, srv1258905363=13, srv342628156=62, srv914424623=92, srv850753440=86, srv1258748155=12, srv1739762035=35, srv2010283524=48, srv2061582561=51, srv609235308=72, srv1150332352=9, srv80825674=82, srv9364796=94, srv979191340=97, srv1472867013=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-19 13:53:17,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-19 13:53:17,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-19 13:53:17,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-19 13:53:17,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-19 13:53:17,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-19 13:53:17,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-19 13:53:17,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-19 13:53:17,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-19 13:53:17,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-19 13:53:17,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-19 13:53:17,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-19 13:53:17,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-19 13:53:17,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-19 13:53:17,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-19 13:53:17,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-19 13:53:17,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-19 13:53:17,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-19 13:53:17,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-19 13:53:17,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-19 13:53:17,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-19 13:53:17,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-19 13:53:17,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-19 13:53:17,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-19 13:53:17,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-19 13:53:17,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-19 13:53:17,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-19 13:53:17,524 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006662924501529447); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010005610098618013); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006662924501529447); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:53:17,524 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-19 13:53:17,524 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0010299234134065612, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006662924501529447); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010005610098618013); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006662924501529447); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-19 13:53:17,863 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 449 ms to try 5000 different iterations. Found a solution that moves 261 regions; Going from a computed imbalance of 0.0010299234134065612 to a new imbalance of 6.0408314513916127E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0025070047769678496); MoveCostFunction : (multiplier=7.0, imbalance=0.029, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:53:17,864 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1020466726:83 , srv1258748155:83 , srv1739762035:83 , srv1862675081:83 , srv32448664:83 , srv883319831:83 , srv1114253798:84 , srv1130862155:84 , srv1135440549:84 , srv1258905363:84 , srv1360245195:84 , srv1364203954:84 , srv1448235260:84 , srv1472867013:84 , srv1486844457:84 , srv1544750042:84 , srv1624626063:84 , srv1676370661:84 , srv1743935831:84 , srv1831215936:84 , srv1904315277:84 , srv1937248082:84 , srv1938550787:84 , srv2010283524:84 , srv2061582561:84 , srv2072570318:84 , srv2073414701:84 , srv2082024426:84 , srv2119650479:84 , srv2122898565:84 , srv2133297157:84 , srv342628156:84 , srv44441552:84 , srv540989264:84 , srv555310386:84 , srv628884539:84 , srv722106953:84 , srv7413056:84 , srv743240987:84 , srv823453102:84 , srv850753440:84 , srv862108761:84 , srv885816629:84 , srv888661502:84 , srv914424623:84 , srv992194527:84 , srv1019776688:85 , srv1030923475:85 , srv1100818659:85 , srv1102697018:85 , srv1150332352:85 , srv117156483:85 , srv1306599683:85 , srv1312385521:85 , srv1363664119:85 , srv140998937:85 , srv1438651312:85 , srv1480802206:85 , srv1547258889:85 , srv1605533195:85 , srv1619763671:85 , srv1685147705:85 , srv1736494758:85 , srv1755037149:85 , srv182277395:85 , srv1935637408:85 , srv1945580515:85 , srv1995346797:85 , srv2010838139:85 , srv2014843238:85 , srv234003506:85 , srv321704973:85 , srv429027790:85 , srv50187028:85 , srv5063628:85 , srv516547335:85 , srv560786850:85 , srv729898444:85 , srv748430389:85 , srv761538373:85 , srv80825674:85 , srv848342604:85 , srv869227296:85 , srv927426340:85 , srv9364796:85 , srv967567616:85 , srv979191340:85 , srv98286284:85 , srv1071065244:86 , srv1257544769:86 , srv1467967424:86 , srv1471578074:86 , srv1794719650:86 , srv238220953:86 , srv568762576:86 , srv737583930:86 , srv819853020:86 , srv952378590:86 , srv609235308:87 , srv801627502:625 } 2023-12-19 13:53:17,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-19 13:53:17,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv748430389=79, srv1862675081=41, srv182277395=39, srv729898444=75, srv1743935831=36, srv1448235260=21, srv722106953=74, srv1130862155=7, srv2122898565=56, srv1071065244=3, srv1605533195=29, srv1312385521=15, srv98286284=98, srv1794719650=38, srv1114253798=6, srv2082024426=54, srv862108761=87, srv1364203954=18, srv1685147705=33, srv848342604=85, srv1547258889=28, srv2073414701=53, srv819853020=83, srv1619763671=30, srv50187028=65, srv888661502=91, srv761538373=80, srv952378590=95, srv1624626063=31, srv32448664=61, srv568762576=71, srv321704973=60, srv1257544769=11, srv1676370661=32, srv1486844457=26, srv238220953=59, srv516547335=67, srv885816629=90, srv1755037149=37, srv1020466726=1, srv1935637408=43, srv1736494758=34, srv5063628=66, srv2072570318=52, srv1471578074=23, srv1831215936=40, srv1467967424=22, srv743240987=78, srv869227296=88, srv429027790=63, srv1904315277=42, srv1019776688=0, srv2133297157=57, srv234003506=58, srv992194527=99, srv1360245195=16, srv1995346797=47, srv823453102=84, srv628884539=73, srv2119650479=55, srv44441552=64, srv883319831=89, srv1945580515=46, srv1480802206=25, srv555310386=69, srv2010838139=49, srv2014843238=50, srv1938550787=45, srv1438651312=20, srv1100818659=4, srv1135440549=8, srv967567616=96, srv1937248082=44, srv801627502=81, srv1102697018=5, srv737583930=76, srv560786850=70, srv1544750042=27, srv1306599683=14, srv540989264=68, srv117156483=10, srv140998937=19, srv1363664119=17, srv927426340=93, srv1030923475=2, srv7413056=77, srv1258905363=13, srv342628156=62, srv914424623=92, srv850753440=86, srv1258748155=12, srv1739762035=35, srv2010283524=48, srv2061582561=51, srv609235308=72, srv1150332352=9, srv80825674=82, srv9364796=94, srv979191340=97, srv1472867013=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-19 13:53:17,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-19 13:53:17,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-19 13:53:17,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-19 13:53:17,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-19 13:53:17,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-19 13:53:17,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-19 13:53:17,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-19 13:53:17,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-19 13:53:17,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-19 13:53:17,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-19 13:53:17,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-19 13:53:17,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-19 13:53:17,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-19 13:53:17,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-19 13:53:17,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-19 13:53:17,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-19 13:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-19 13:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-19 13:53:17,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-19 13:53:17,992 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0025070047769678496); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:53:17,993 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-19 13:53:17,993 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.858075345267032E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0025070047769678496); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-19 13:53:18,214 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 320 ms to try 5000 different iterations. Found a solution that moves 43 regions; Going from a computed imbalance of 5.858075345267032E-5 to a new imbalance of 7.68884993569022E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0047777777777777775); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:53:18,215 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1030923475:83 , srv1306599683:83 , srv1467967424:83 , srv1794719650:83 , srv5063628:83 , srv568762576:83 , srv609235308:83 , srv737583930:83 , srv952378590:83 , srv1019776688:84 , srv1071065244:84 , srv1100818659:84 , srv1102697018:84 , srv1114253798:84 , srv1130862155:84 , srv1135440549:84 , srv1150332352:84 , srv117156483:84 , srv1257544769:84 , srv1258748155:84 , srv1312385521:84 , srv1360245195:84 , srv1363664119:84 , srv1364203954:84 , srv140998937:84 , srv1438651312:84 , srv1448235260:84 , srv1471578074:84 , srv1472867013:84 , srv1480802206:84 , srv1486844457:84 , srv1544750042:84 , srv1547258889:84 , srv1605533195:84 , srv1619763671:84 , srv1624626063:84 , srv1676370661:84 , srv1685147705:84 , srv1736494758:84 , srv1739762035:84 , srv1743935831:84 , srv1755037149:84 , srv182277395:84 , srv1831215936:84 , srv1904315277:84 , srv1935637408:84 , srv1937248082:84 , srv1945580515:84 , srv1995346797:84 , srv2010838139:84 , srv2014843238:84 , srv2061582561:84 , srv2072570318:84 , srv2073414701:84 , srv2082024426:84 , srv2119650479:84 , srv2122898565:84 , srv2133297157:84 , srv234003506:84 , srv238220953:84 , srv321704973:84 , srv32448664:84 , srv342628156:84 , srv429027790:84 , srv50187028:84 , srv516547335:84 , srv540989264:84 , srv555310386:84 , srv560786850:84 , srv722106953:84 , srv729898444:84 , srv743240987:84 , srv748430389:84 , srv761538373:84 , srv80825674:84 , srv819853020:84 , srv823453102:84 , srv848342604:84 , srv850753440:84 , srv862108761:84 , srv869227296:84 , srv883319831:84 , srv885816629:84 , srv888661502:84 , srv914424623:84 , srv927426340:84 , srv9364796:84 , srv967567616:84 , srv979191340:84 , srv98286284:84 , srv1020466726:85 , srv1258905363:85 , srv1862675081:85 , srv1938550787:85 , srv2010283524:85 , srv44441552:85 , srv628884539:85 , srv7413056:85 , srv992194527:85 , srv801627502:684 } 2023-12-19 13:53:18,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-19 13:53:18,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv748430389=79, srv1862675081=41, srv182277395=39, srv729898444=75, srv1743935831=36, srv1448235260=21, srv722106953=74, srv1130862155=7, srv2122898565=56, srv1071065244=3, srv1605533195=29, srv1312385521=15, srv98286284=98, srv1794719650=38, srv1114253798=6, srv2082024426=54, srv862108761=87, srv1364203954=18, srv1685147705=33, srv848342604=85, srv1547258889=28, srv2073414701=53, srv819853020=83, srv1619763671=30, srv50187028=65, srv888661502=91, srv761538373=80, srv952378590=95, srv1624626063=31, srv32448664=61, srv568762576=71, srv321704973=60, srv1257544769=11, srv1676370661=32, srv1486844457=26, srv238220953=59, srv516547335=67, srv885816629=90, srv1755037149=37, srv1020466726=1, srv1935637408=43, srv1736494758=34, srv5063628=66, srv2072570318=52, srv1471578074=23, srv1831215936=40, srv1467967424=22, srv743240987=78, srv869227296=88, srv429027790=63, srv1904315277=42, srv1019776688=0, srv2133297157=57, srv234003506=58, srv992194527=99, srv1360245195=16, srv1995346797=47, srv823453102=84, srv628884539=73, srv2119650479=55, srv44441552=64, srv883319831=89, srv1945580515=46, srv1480802206=25, srv555310386=69, srv2010838139=49, srv2014843238=50, srv1938550787=45, srv1438651312=20, srv1100818659=4, srv1135440549=8, srv967567616=96, srv1937248082=44, srv801627502=81, srv1102697018=5, srv737583930=76, srv560786850=70, srv1544750042=27, srv1306599683=14, srv540989264=68, srv117156483=10, srv140998937=19, srv1363664119=17, srv927426340=93, srv1030923475=2, srv7413056=77, srv1258905363=13, srv342628156=62, srv914424623=92, srv850753440=86, srv1258748155=12, srv1739762035=35, srv2010283524=48, srv2061582561=51, srv609235308=72, srv1150332352=9, srv80825674=82, srv9364796=94, srv979191340=97, srv1472867013=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-19 13:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-19 13:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-19 13:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-19 13:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-19 13:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-19 13:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-19 13:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-19 13:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-19 13:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-19 13:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-19 13:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-19 13:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-19 13:53:18,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-19 13:53:18,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-19 13:53:18,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-19 13:53:18,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-19 13:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-19 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-19 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-19 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-19 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-19 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-19 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-19 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-19 13:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-19 13:53:18,359 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:53:18,359 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1030923475:83 , srv1306599683:83 , srv1467967424:83 , srv1794719650:83 , srv5063628:83 , srv568762576:83 , srv609235308:83 , srv737583930:83 , srv952378590:83 , srv1019776688:84 , srv1071065244:84 , srv1100818659:84 , srv1102697018:84 , srv1114253798:84 , srv1130862155:84 , srv1135440549:84 , srv1150332352:84 , srv117156483:84 , srv1257544769:84 , srv1258748155:84 , srv1312385521:84 , srv1360245195:84 , srv1363664119:84 , srv1364203954:84 , srv140998937:84 , srv1438651312:84 , srv1448235260:84 , srv1471578074:84 , srv1472867013:84 , srv1480802206:84 , srv1486844457:84 , srv1544750042:84 , srv1547258889:84 , srv1605533195:84 , srv1619763671:84 , srv1624626063:84 , srv1676370661:84 , srv1685147705:84 , srv1736494758:84 , srv1739762035:84 , srv1743935831:84 , srv1755037149:84 , srv182277395:84 , srv1831215936:84 , srv1904315277:84 , srv1935637408:84 , srv1937248082:84 , srv1945580515:84 , srv1995346797:84 , srv2010838139:84 , srv2014843238:84 , srv2061582561:84 , srv2072570318:84 , srv2073414701:84 , srv2082024426:84 , srv2119650479:84 , srv2122898565:84 , srv2133297157:84 , srv234003506:84 , srv238220953:84 , srv321704973:84 , srv32448664:84 , srv342628156:84 , srv429027790:84 , srv50187028:84 , srv516547335:84 , srv540989264:84 , srv555310386:84 , srv560786850:84 , srv722106953:84 , srv729898444:84 , srv743240987:84 , srv748430389:84 , srv761538373:84 , srv80825674:84 , srv819853020:84 , srv823453102:84 , srv848342604:84 , srv850753440:84 , srv862108761:84 , srv869227296:84 , srv883319831:84 , srv885816629:84 , srv888661502:84 , srv914424623:84 , srv927426340:84 , srv9364796:84 , srv967567616:84 , srv979191340:84 , srv98286284:84 , srv1020466726:85 , srv1258905363:85 , srv1862675081:85 , srv1938550787:85 , srv2010283524:85 , srv44441552:85 , srv628884539:85 , srv7413056:85 , srv992194527:85 , srv801627502:684 } 2023-12-19 13:53:18,387 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=286 (was 276) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=3118 (was 3717) 2023-12-19 13:53:18,396 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=286, ProcessCount=170, AvailableMemoryMB=3117 2023-12-19 13:53:18,397 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-19 13:53:18,397 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-19 13:53:18,405 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1078643428:84 , srv1134332977:84 , srv1139509369:84 , srv1150002957:84 , srv117180863:84 , srv1196875340:84 , srv1286369324:84 , srv1367067044:84 , srv1604624544:84 , srv1697079226:84 , srv1750597056:84 , srv1808084620:84 , srv1894878808:84 , srv205783259:84 , srv2069200532:84 , srv212812995:84 , srv239749605:84 , srv244895136:84 , srv402970163:84 , srv40532500:84 , srv480744077:84 , srv526065192:84 , srv534458688:84 , srv610256236:84 , srv751236232:84 , srv786700138:84 , srv809916932:84 , srv836563818:84 , srv964747116:84 , srv1763120282:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-19 13:53:18,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-19 13:53:18,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1894878808=13, srv1078643428=0, srv1763120282=11, srv205783259=14, srv2069200532=15, srv751236232=25, srv1604624544=8, srv786700138=26, srv402970163=19, srv1750597056=10, srv239749605=17, srv836563818=28, srv1697079226=9, srv117180863=4, srv212812995=16, srv1196875340=5, srv809916932=27, srv1134332977=1, srv610256236=24, srv1150002957=3, srv480744077=21, srv964747116=29, srv1139509369=2, srv1286369324=6, srv40532500=20, srv244895136=18, srv526065192=22, srv534458688=23, srv1808084620=12, srv1367067044=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-19 13:53:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-19 13:53:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-19 13:53:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-19 13:53:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-19 13:53:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-19 13:53:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-19 13:53:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-19 13:53:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-19 13:53:18,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-19 13:53:18,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-19 13:53:18,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-19 13:53:18,435 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1135295576693174, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:53:18,436 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912550771193415, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1135295576693174, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-19 13:55:18,420 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6078845 different iterations. Found a solution that moves 1850 regions; Going from a computed imbalance of 0.9912550771193415 to a new imbalance of 4.317992290299789E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6851851851851852, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:55:18,426 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1078643428:90 , srv1134332977:90 , srv1139509369:90 , srv1150002957:90 , srv117180863:90 , srv1196875340:90 , srv1286369324:90 , srv1367067044:90 , srv1604624544:90 , srv1697079226:90 , srv1750597056:90 , srv1763120282:90 , srv1808084620:90 , srv1894878808:90 , srv205783259:90 , srv2069200532:90 , srv212812995:90 , srv239749605:90 , srv244895136:90 , srv402970163:90 , srv40532500:90 , srv480744077:90 , srv526065192:90 , srv534458688:90 , srv610256236:90 , srv751236232:90 , srv786700138:90 , srv809916932:90 , srv836563818:90 , srv964747116:90 } 2023-12-19 13:55:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-19 13:55:18,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1894878808=13, srv1078643428=0, srv1763120282=11, srv205783259=14, srv2069200532=15, srv751236232=25, srv1604624544=8, srv786700138=26, srv402970163=19, srv1750597056=10, srv239749605=17, srv836563818=28, srv1697079226=9, srv117180863=4, srv212812995=16, srv1196875340=5, srv809916932=27, srv1134332977=1, srv610256236=24, srv1150002957=3, srv480744077=21, srv964747116=29, srv1139509369=2, srv1286369324=6, srv40532500=20, srv244895136=18, srv526065192=22, srv534458688=23, srv1808084620=12, srv1367067044=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-19 13:55:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-19 13:55:18,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-19 13:55:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-19 13:55:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-19 13:55:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-19 13:55:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-19 13:55:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-19 13:55:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-19 13:55:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-19 13:55:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-19 13:55:18,443 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-19 13:55:18,444 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1078643428:90 , srv1134332977:90 , srv1139509369:90 , srv1150002957:90 , srv117180863:90 , srv1196875340:90 , srv1286369324:90 , srv1367067044:90 , srv1604624544:90 , srv1697079226:90 , srv1750597056:90 , srv1763120282:90 , srv1808084620:90 , srv1894878808:90 , srv205783259:90 , srv2069200532:90 , srv212812995:90 , srv239749605:90 , srv244895136:90 , srv402970163:90 , srv40532500:90 , srv480744077:90 , srv526065192:90 , srv534458688:90 , srv610256236:90 , srv751236232:90 , srv786700138:90 , srv809916932:90 , srv836563818:90 , srv964747116:90 } 2023-12-19 13:55:18,457 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=183 (was 286), ProcessCount=167 (was 170), AvailableMemoryMB=2168 (was 3117)